I would also note that in case of delayed flight/IRROPS, I would take any GA projection of time to be a guideline, not a promise. Fact is, when someone says 20 minutes, they are saying we're going to do this ASAP and my estimate is that we'll be ready around then. They are not saying it will be at that exact time. Also, as noted, OP stated that they started back to gate at 20 minute mark. Personally, unless I was indifferent to making the flight (which the OP wasn't), I would stay within sight of the gate.