Let them notice. And if you are traveling on a pass, let them give you grief.
As for me, I pay for J largely so I can travel in greater comfort. I don't know how I would respond if a GA ever made me feel uncomfortable for travelling in sweats, but it wouldn't be pretty.
(sorry, I know this is totally off topic)
Employees traveling on passes have dress codes that are FAR more strict than revenue pax.
You'd have to be missing an important article of clothing before they'd say anything to you.