I've strictly answered the OP's question without opining about their plan to visit downtown.

I will now opine about visiting downtown given OP's schedule.

I agree with the others above regarding OP's plan to visit downtown. It will be tight, at best. I might consider doing it for a domestic-domestic connection, but that's still not much time to get downtown, see/do something, then get back in time for last boarding call. But hey, I've done it before myself with only 3.5 hours between flights, but that was to go back to my place to pick something up and then head back to YYZ before going onward overseas. And even with NEXUS/status, it was still tight without traffic/delays getting to/from downtown, and I didn't have to preclear US customs since I was going TATL.