I can't stand walking slowly. I don't care what they think, I will walk past them as long as there is ample room to do so without bumping into them. I have done this when boarding planes multiple times, and whenever I look behind me, others have also passed the slow walker.
When you stop at the door I'll have caught up to the fast walkers. I hope the few seconds you save are worth it.