Indeed, this code assumes that Maps are always statically specified value Maps of reasonably small size.
I've optimized this analogously to our array conversion optimizations in 2.0.1: We check in advance whether there is any point in trying to convert elements, checking for generic key/value types as well as for custom editors for this particular collection. This means that we don't even try to create a typed collection, avoiding to iterate over the collection's original elements just to find out that no element had to be converted in the first place.
So in your scenario, as long as the target property is not declared as typed Map but rather just as plain Map, no conversion will be attempted, hence no iteration of the original entries will happen. Note, however, that a Map<String, Object> declaration (or the like) would still trigger a check of all keys... Consider declaring your receiving property/argument as plain Map in this case, internally accessing it as typed Map through a cast.
This refinement will be available in the next 2.0.5 snapshot (http://www.springframework.org/snapshots). Please give it a try and let us know whether it works for you!