You are DEFINITELY spending too much time in Toronto. 4 nights is more than enough. I agree with a previous poster that you should take the train or fly to Montreal for at least 3 nights and try to fit in Quebec City for 2 nights. It's really a waste to spend all that time in Toronto. There's really not that much to see there. In fact, Montreal and Quebec City are much more beautiful and culturally rich.

As far as Calgary is concerned, you should drive to the rockies and spend time in Banff/Lake Louise area. A really gorgeous area with lots to do. Calgary, in my opinion, is a bore.