If you do Harbor Air from Vancouver downtown to Victoria downtown, there are other ways to get to Butchart Garden.
Google the website, this is my first post so I can't post URL.
Seattle may be hard on the driver because the 3 hour drive each way. That's assuming you don't get hold up at the border (other people or paying the $6USD / person entry fee). If you go on a weekday, expect long delays on Interstate 5 between Seattle and Marysville during morning and afternoon rush hours. Weekends, expect 1 hour wait at the car land crossing.
I suggest dropping Calgary and replace with Alberta Dinosaur Park where you get to see the real fossils.
I would also drop Seattle unless you plan to stay over night. In Seattle, I suggest the following:
* Pike Market, default tourist trap, horrid traffic and parking
* Museum of Flight, it's south of downtown Seattle
* Chihuly Garden and Glass, if you're into art
* Museum of Pop Culture, if you're into music and it's very close to the Chihuly Garden
* Boeing Factory in Everett, it's between Vancouver and Seattle