If you do want to visit Victoria while you're here, I recommend Harbour Air. They fly directly from Vancouver Harbour to Victoria Harbour in 25 minutes, so it makes for an easy day trip and can be had for as little as $99 each way if you book ahead. The ferry trip is nice, but pretty much an all-day affair because the endpoints (Tsawwassen and Swartz Bay) are so inconvenient.