Yeah, that might be a problem. I don't take my glasses off in any public place where there is any remote chance of flying debris, which pretty much means any public place. The multiple ophthalmologists who treat and monitor my conditions would not approve of me doing so. I'm not even supposed to use standard plastic lenses for daily use because they do not have enough shatter resistance.

Any border agent or TSA agent who insists on me removing my glasses will not get any cooperation. It ain't going to happen unless they arrest me and do it themselves, and then there's going to be an endless number of complaints.