Depends also what your plans are after landing in FRA. Continuing to France by train, plane, rented car?

I agree that the train station area is rather off-putting, but Frankfurt itself does have some nice hotels, like Grandhotel Hessischer Hof (pricey, but worth it) and Villa Orange, to name just two. Museums are plenty and interesting. There's a direct train connection to Paris too (3hrs).

Mainz and Wiesbaden are prettier than Frankfurt architecture- and atmosphere-wise, and both reachable by suburban train from the airport, as mentioned above.
Cologne is too far for my taste, and (apart from the cathedral) not any nicer than Frankfurt (both cities were bombed heavily during WWII, so most architecture is from the 1960s/70s).
The Rhine valley is nice, but imho better viewed from a ship - ships leave from Mainz, Wiesbaden, Rüdesheim etc.
Rüdesheim is a cute little town, but veeery touristy.