Even generously allowing 1.5 hours each way by public transit to be back at the airport an hour before the flight still leaves you almost 5 hours of free time in the city.

The key I think is having a list of things you are going to do, so that you are comfortable with your timing.

Summer is a great time to visit Toronto. In addition to the "always there" attractions, there are tons of events all over, to suit your taste