Unless you have something specific to see in Calgary I'd drop it for an overnight somewhere on the way back to Vancouver (Kamloops or Kelowna). That being said I agree with all of the above re: driving to Banff/Lake Louise. I've done Calgary-Kelowna a few times and it can be ok but when there's a large chunk of traffic - especially 18-wheelers and RVs - it can be tedious.

I would consider sticking to the west coast and exploring more of the Whistler area and Vancouver Island. Victoria is a great little city and it's easy to do a day trip to Nanaimo or an overnight trip to Tofino to check out that part of the island too.