It's very likely that the pilot had no idea him/her self. First, I'm sure they were dealing with the paperwork necessary after removing someone from a flight. Second, it's not as easy as saying "ok, got that guy off, now we can take off...". They would have to get a new departure slot (and arrival slot) and maybe file a new flightplan, remove the passengers bags, recheck weight/balance, etc. etc. In other words, plenty to keep them busy. I would say that 20 minutes is acceptable!