Depending how small you want, ski towns are usually a great option in the off season (as mentioned up thread).
We once visited Yellowstone in June and spent quite a bit of time in the surrounding towns. Weather was still a bit iffy, but lots of variety between towns in terms of size, culture, etc., and most importantly, still early enough for low numbers of tourists.
Boise doesn't pop up on many peoples radar, but I found it a charming small city. Likewise for Austin.
Nashville ticks your boxes, and usually not too hot yet in June. Several daily nonstop flights from LAX, too.
Also, I know you said you wanted to get out of Cali, but it's a great time of year to do a leisurely road trip through the gold country. A long string of small, walkable towns along route 49.