I was recently at a carrier's lounge at another airport where it was overcrowded and the only available space was additional temporary seating within earshot of the reception area. I was surprised that what seemed to be more than half of the people who showed up at the counter were ineligible for entry.
Before I left, I asked one of the agents if this was common and she told me that not only was it common, but that the excuses were constant from people who clearly had no right to enter.
No one exception ever matters. In fact, had the agent OP encountered allowed him full access for the entire day it would not have mattered. The problem is that if one gives in to one request, what about the next one and the one after that? Maybe the rule ought to be that one has arrivals access for bathroom use only. But, it is not.