I agree that Harajuku and Azabu-Juban are going to be optimal locations for your tastes. Ebisu and Ginza could also be worth exploring.
Shinjuku and Shibuya are both sort of mass-market commuter hubs, very much Midtown-like. Roppongi is where you go to vomit Midori sours onto the street and get hustled by Nigerian strip club touts.