I'm a little surprised at the answers so far - not that the suggestions aren't good, but that many of the cities mentioned don't fit my definition of "small or medium" (I do understand that each of us will have his/her own feeling on this).

I think of "medium" as maybe 50,000 to 300,000 population, and "small" as under 50k population.

For "interesting", the first place that came to my mind is Columbus, Indiana, population about 40,000. What distinguishes this town from any rural Midwestern county seat is its architecture - the CEO of Cummins Engine Co. (Cummins Diesel) instituted a program under which the company paid architect's fees for public and private buildings, resulting in schools, churches, government and commercial buildings from the likes of Eero Saarinen, I.M. Pei, Robert Venturi, Cesar Pelli, Richard Meier and others.

Visiting the town, it's interesting how the architectural works are integrated into the framework of the town as a whole - these are not sterile, stand-offish structures.