Three or four days is enough to cover Madrid. Instead of moving around, you might consider taking day trips. Segovia and Toledo are easy one day outings from Madrid using trains. I did not get to Toledo but be forewarned that if you go to Segovia, don't go on a Monday; pretty much everything but the castle is closed then.