This has not been my experience, although many times, especially at YYZ, boarding is delayed. A number of times at YHZ, I've come down from the MLL about 5-10 mins after the scheduled boarding time to find that there only a couple of people left to board.

But your point is well-taken. It's extremely frustrating to hang around the gate without being given any information about when the boarding will actually commence.