Very limited experience in Kazakhstan and Armenia -- about one week each -- but Georgia is one of my favorites. The food and wine are great, and Tbilisi's a pretty nifty city, combining centuries-old, Soviet, and contemporary design in a bustling though compact place.
In Kazakhstan, I was in Almaty and Astana (now Nur-Sultan). Not much to do in Almaty, besides a ropeway, but Astana had some bizarre architecture. Lots of rip-offs in that country, based on my experience bargaining and taxiing. I've heard Charyn Canyon is nice.
In Armenia, I centered on Yerevan, taking day trips to Garni, and monasteries. Good kebabs and lahvash.