For fish, I've heard good things about Ray's in the City. I went to the sister restaurant, Ray's on the River, years and years ago, and enjoyed it a great deal. Then again, I'm not a particularly picky eater.
Then there's Mary Mac's Tea Room. I've not been in years, again, but it's pretty much a standard place for folks visiting Atlanta to have a Southern meal. You can get things like fried green tomatoes, hoppin' john, and collard greens and cracklin'. Now, this seems normal to me, but I remember the time I went my northern visitors weren't used to them at all.