Have you been to Duke Point? It's a pretty unpleasant terminal. Unlike Departure Bay, which has a kid's play, big food court, views of seaplanes etc. Duke Point is a bit of a wasteland.

In terms of distance, Horseshoe Bay is 37 km from YVR, Tswwassen is 29K so distance difference is marginal.

There are 8 departures per day out of Tsawwassen and 8 out of Horseshoe Bay. The Tsawwassen <--> Duke Point crossing is 25 minutes longer and less scenic.

...but in the end it really comes down to whether Departure Bay or Duke Point is more convenient.