As far as I remember, it's just a ferry terminal for most people. There are some marinas there. Other than nearby businesses hoping to extract cash out of those with long ferry waits, I doubt there is too much to do there (hasn't worked at Departure Bay in Nanaimo). I wouldn't go to Swartz Bay just for the day. If anything, keep on going to Victoria. The PCL coach driver sells tickets (assuming the bus isn't full).

And how are you getting to Tsawwassen? It's in the middle of nowhere too.