I travelled for many years without my card and never had a problem.

On the odd occasions in the US I was asked for it, I politely explained I didn't have it on me and invariably they would tap away a bit, smile and welcome me in.

Don't lose any sleep over it.