Some of the best small towns in the US happen to be college towns. If you're willing to visit on a gameday and join in the fun, you can have a really amazing weekend experience...

For my money, the best college town in the country is Lawrence, KS. I'm no KU fan - far from it - but Lawrence is a gorgeous downtown to walk around with gobs of character. A very close second is Madison, WI - which has already been repeatedly mentioned. Both have lots of local shops and restaurants, a walkable urban core, and arts/music venues of caliber disproportionate to their size.

For second tier medium-to-large cities, I second the Seattle, Portland, and Portland recommendations many have made...