I've not had to guest someone in myself but in reading some of the stories where the lounge dragons have changed their minds, I also wonder how many other passengers were around at the time? Do they reject if there's people milling about, so as not to encourage unwanted behaviours, and then wave you in when coast is a bit clearer?