My partner has once been denied access using BA Gold card with only his QF Silver number on the boarding pass. A real lounge dragon in this instance.
So he changed it to his BA number and then changed it back before the flight boarded. Crazy.
This was a couple of years ago now.