In a way, it's kind of obvious the average person who doesn't fly often doesn't know about ff programs and the privileges that come with them, priority boarding being one of them.
I think it's fair game when someone questions me why I'm cutting in line and board earlier than him/her.
It happens, and I learned to live with it.