Maybe not to you, maybe not to me, maybe not to anyone here, but in that's a judgment call on the cabin crew's part. I've seen FAs ask one of two people waiting for the lav to go back to his or her seat until nobody else is waiting for it. In their opinion at that time, two was congregating - and theirs is the only opinion that matters.