The Prado museum is worth a visit but often VERY crowded on weekends...there is a lot of very traditional art there and if you are looking for anything modern or cutting edge you would be better off at the Reina Sofia.

Prada is a pricey boutique just off the Calle Serrano on Goya

I like Toledo much more than Avila or Segovia...however they were building a new train station last year at Toledo and I doubt it is complete. If that is the case, your "train" ride will involve a transfer to a bus, which you will no doubt want to avoid.

The AVE is a really nice experience, and the weather in Sevilla will be great in Nov. so that would be a good option...but since your stay is so short I would really just stay in Madrid and enjoy your time there rather than worry about getting somewhere else.