I have only recently started carrying my card. I travelled for many years without out and had no issues getting access to lounges, in the US and elsewhere.

Occasionally I've been asked for the card, but when I explained I didn't have it, the lounge agent tapped away at something or other and I was soon waved in.

Your BP should be your ticket. I wouldn't worry about it.