This thread got me to start thinking about all the cities I've been to. My short list that I would go back to are:
Medium:
Portland, OR
Victoria, BC
Charleston, SC
Savannah, GA
Spokane, WA
Small:
Walla Walla, WA
Dayton, OH
Cheyenne, WY
Halifax, NS (or is it medium.. I don't care, I'd go back!)
Estes Park, CO (it's a bit of a drive from Denver, but if you go to the Scottish Festival it is awesome!)