I'm of the opposite opinion; Gaoxiong is way more photogenic, if anything just because of the way it's situated on the sea. There're a lot of formerly industrial areas redeveloped in an interesting way plus some excellent city parks, and the Love River boulevards are great for a walk too. In a number of ways it has actually overtaken Taipei, which is quite an achievement considering the disproportionate amount of funding that goes to the capital. Further, what might matter to the OP is that it's easy to get around on the public transport.

On the other hand, food aside, Tainan is a lot of temples plus the overrated Fort Zeelandia. Everything else looks just like any other town in Taiwan: if you've seen one of them, you've seen them all. Even if the OP had the chance to visit another city, I'd recommend Taidong, Jiayi or (parts of) Zhanghua instead. Come to think of it, if the OP opts for taking the HSR to Taizhong, he could actually rent a car there and see places like Mt Bagua or the Longshan Temple in Lugang, perhaps doing some oyster fishing in Wanggong as well. Another option would be to take a branch line train all the way to Checheng. Either option feels more fun than staying in Taizhong to me.

As for the food, it ranges from great to mediocre everywhere, it's honestly a crapshoot, mostly (except seafood) regardless of the price bracket. If you can read Chinese, check out the opinions on sites like ipeen.com.tw or at least Google reviews, and remember to take them with a grain of salt to account for all the astroturfing and people who are easily pleased (particularly so for foreign cuisine).

Question to the OP: have you been to any other places in Taiwan besides Taipei thus far? Which ones did you like and dislike, and why? Perhaps this way the suggestions can be more suited to your personal preferences.

Hope you have a nice trip.