If you want to check out the campus of U of T and with 4 people my advice is to take a cab right to Bloor and University Avenue(Harbord and St.George to be right in the centre of it)
http://www.torontopearson.com/en/toandfrom/taxilimo/#
If you click the link for the pdf East of the Airport will give you the set price into that area. You definitely have time for that and probably a quick meal in the Yorkville area. Flagging a cab on the way back to the airport will be very easy. CN Tower on a summer Saturday I really don't think would be feasible. You will definitely get a full meal in J on AC880.