Tangent #(n+1): on golf forums the topic of shorts (and white belts) pops up frequently. Quite a large proportion of golfers thinks no man over the age of twelve should be seen in shorts on public - beaches excepted. And yet, most golfers I know almost always play in shorts.
Ergo: wear what you want, nobody really cares.
Tangent #(n+2): try rocking up in s suit and a tie in a "we don't care what you wear, anything you are comfy in is fine" workplace and watch the waves of shock and awe go around. Thought you guys don't care...
Ergo: maybe they care after all but if and only if dressing up.