Not quite the same but I got hassled once using my Passport Card as ID at a TSA checkpoint. For some reason he took exception to it -- didn't refuse but said to the guys at the neighboring stands "Hey Joe, look at this" and proceed to walk over and show the other document inspectors. He clearly was waiting for me to show some other form of ID. I just kept telling him it was an official US Government document and finally he relented. The passengers backed up behind me were getting restless and I probably would have folded before too long.

This was when there were stories of agents copying address and rifling thru passports looking at stamps.