Depending on your time-money tradeoffs, I'd look into leaving your car on the mainland and renting in Victoria. You could save enough time to justify the expense, especially when you consider the car's ferry tickets.

Slightly off-topic story from a few years back: I was flying from Victoria to SEA. Pilot said we were going to make an unscheduled stop in Bellingham, Wash. Reason: Victoria didn't have US customs facilities to pre-clear passengers, so we'd have to clear in Seattle. If we did that, we'd be in line behind three packed 747s just in from Asia. This way, we landed in Bellingham, woke up the customs guy, did our thing in two minutes, and arrived in SEA as domestic!