It's dependent on what you want to do and how much you want to spend. Just to ride on previous posters' comments, I'd probably recommend staying in Vancouver unless you've been here before. I think there's enough places to visit for four days..

Whister still has events happening during the summer but they're typically pretty minor. (Obviously, recommend checking an events calendar.) The stops along the way (Squamish, Britannia Beach/Mines, Shannon Falls, etc) might be of interest to the family also. One caveat though is the highway is a potential PITA. An accident or constuction could snarl up traffic so that you end up crawling or stopped along the highway.

Victoria is nice too but what prevents me from heading over more often is the ferry prices for a car & passengers. It might be worth checking out a coach tour if you want to daytrip to Victoria.