I think the main problem is that there are no great restaurants or things to see near the airport. That means your options are to either stay at the airport, or take the UPX downtown. It's about 25 minutes on the train, and they leave every 15 minutes. Add a few minutes to walk to the station, and say it takes you 45 minutes in each direction. That's 1.5 of your 5.5 hours gone. If you planned to be back at the airport 1.5 hours ahead of your return flight, that leaves 2.5 hours. Time to deplane and getting through Canadian customs can vary from day-to-day. Add in the time to walk from Union Station to your chosen restaurant, wait for a table, and you don't have a lot of time left to enjoy the dinner.