I've done it a couple of times (and would do it again in similar circumstances)

Once was when, having sat at the gate at AMS for about ten minutes, we were informed of a two-hour flight delay. We had been exchanging a few words with the couple sitting next to us; and the words "we're going back to the lounge - would you like to join us?" seemed to come quite naturally.

The other time was a few years ago, just a couple of days after KLM had stopped lounge access for Silver Elites. I was sitting quite near to the reception desk at NWI when someone tried to get in using a Silver card. On overhearing his conversation with the (very apologetic) receptionist I just felt I wanted to be nice, so I turned round and asked her if she would let him in as my guest.