I can't stand walking slowly. I don't care what they think, I will walk past them as long as there is ample room to do so without bumping into them. I have done this when boarding planes multiple times, and whenever I look behind me, others have also passed the slow walker.
Why don't you wait until the jetway/aisle are clear so that you can get the fast walking in without obstacles?