It's all random. I'm often met with strange looks when visiting a lounge different to the one that I "should" be visiting, but I don't recall ever encountering serious resistance. Actually I do recall one occasion, but I was flying Lufthansa on an AZ ticket that had been endorsed because of a cancelled AZ flight, and I guess the AF loungedragon at FCO was right to tell me to go away (I did end up using the Alitalia lounge without any problems).