The rule is T-20 for domestic and transborder, T-30 for international. That is from scheduled departure. There is no requirement for "last call" and they can make the announcement, if one is made, at any time.

For OP, it matters whether he was there at T-19 or T-21. I the former, he was properly off-loaded. If the latter, he was not.

As to delays, the rule is there because they are simply projections and often are reduced or disappear.

Seems harsh, but there has to be a cutoff and on busy flights, it can take time to complete boarding and deal with last minute issues and still push on schedule.