I remember once that BA in JFK would not let me in without the card. I think I was flying some codeshare and while my # was on the boarding pass, there was no status mention. They said I would have to go outside security and to the check-in counters to resolve.
The T7 lounge was terrible so not missing much ... but at least one anecdote for why having the card could be helpful.