Thanks for all the replies.
I've decided to start early morning from Seattle. Stay in Vancouver that night. Catch ferry to Victoria next day morning. Does it make sense to stay in Victoria that night or return to Vancouver? I am getting pretty cheap rates @ Vancouver DT hotels.
If I stay in Victoria, I believe I 'll have to catch another ferry to re-enter into WA state.