I'd have to second the Union Square market in New York. Just went there today for the last time in awhile since I'm moving to the burbs The best time to visit is in the fall when these amazing grapes are sold. Varieties I haven't seen anywhere else.

I went to a cool, small farmer's market in Healdsburg, California in Sonoma County once. Great looking summer fruits. Can you tell I like fruit? My wife has been trying to get me to eat less of it for our entire relationship since I eat fruit so often.

Another one I liked was in Quebec City. Was there in late summer one year when they had fabulous berries.

The best fruit I ever had came from a farmer's store between Yosemite and San Francisco...though it wasn't part of a whole farmer's market just a free-standing store. I can't remember the name of the small town.