I would normally ask them if they are following me into lounges, if I feel that they are not entitled to be in, I would apologize and just close the door and tell them that they should use their own room key to get in.

I don't feel that I am being rude, but just that I don't want these non-eligible people causing issues to us eligible guests in the long term..

Cheers!