We (2 US citizens) did the drive from downtown Seattle (the Hertz HLE) to downtown Vancouver on a Sunday afternoon 2 weeks ago. I'm guessing it was about 4-1/2 hours all in, but we made a Starbucks stop near UW, a grocery/gas run in Bellingham, then a lunch run near Blaine.
The wait at the border was perhaps 15 min.

The drive from the border to downtown Vancouver was a solid 30-45 min. There was tons of traffic, and once the freeway ended, the road was mostly just 2 lanes (a 3rd lane came and went), with no left turn lanes.

We drove up to Whistler for a day, left around 10am from the downtown Hampton Inn on a Tuesday morning, and I'm guessing it took 2 hours. We did a casual drive, stopped at Shannon Falls, then at a Starbucks further up.
The drive back from Whistler to town wasn't bad, until again we got off the highway, there was tons of traffic near a mall, then getting onto the bridge.