What about those F-class flight where they give out silk sleeper suits? Compare that to how you would feel if you saw a pax in coach board wearing his/her flannel pajamas. Our perception of people is influenced by many factors, including what class of service they sit in and there appearance.
That said, the workplace dress code has relaxed much in the last five or ten years. Company CEOs may be walking arround in jeans and tshirts, and not just internet CEOs fresh out of school.
As long as you wear clean clothes that aren't too revealing, (I think socks are a must - we're getting feed, after all) I have no problem with tshirts, shorts if you must (some of these planes get pretty hot at times) regardless of the class of service you sit in.