You can also take a general coach bus (Greyhound, etc) that will take you on the ferry and into Victoria (and back) at pre-arranged times. It's probably the most convenient option.

I've also done the coach bus tour, albeit many years ago. It's a good option to have someone drive you around and have a guide, as well.