I'm glad I'm not the only one. I end up convinced I'm some sort of mini-Hitler about these things.

No, don't rush Sir, you're just holding up a plane with 350 people sitting on it. 350 people who managed to get to the gate in good time so we could board in time for our take-off slot...

I remember talking to a FA a few years back now on BA and she said that it's a race between the passenger and their luggage. If their luggage is located before they are, the luggage is offloaded and they leave without them...