I did a trip to Spain & Portugual a few years back and stopped at both Madrid and Barcelona. Found both cities to be very enchanting but Madrid had a few drawbacks.

One was that English didn't seem to be widely spoken as in Barcelona - wasn't a real problem but noticeable. Second, found the cabbies to be less than honest - trying to double charge me from the airport, adding extra fees and using the language as an excuse. And thirdly, almost had my luggage stolen while arguing with a cabby in front of the hotel and a young kid starts pestering me to take my luggage into the hotel (mostly with sign language). Luckily I said no, because when I went into the hotel and asked about it, they said they had no luggage service!

Just a couple of points to be aware of - didn't lessen my enjoyment of the trip overall but it's those type of things that can make a big difference!