The difference in having the card in your wallet is for when it is needed for access to non-AA lounges. It is often reported here on FT that access is denied if you can't produce the card although fortunately it seems to be becoming less of a problem. Per the OW lounge access policy, presenting the card is a requirement so one can't argue if one is refused access in this instance.