seems that your bitdepth is too low to cover your displacement informations.
Check your search distance (if working in 16 bit space). It should be as close as possible to your highRes surface.
If that doesn`t solve your problem try to extract your details in floating point space.
Mental ray can render both, 16 and 32bit displacement maps.