Flying should be comfortable but please don't show up all scruffy like you just got up from bed. I like to wear dark jeans with no holes and a nice comfortable sweater with brown shoes and have flown in First wearing it with no problems. I also have no problems wearing a suit if that's what the dress code is then I'm not one to challenge it.
Smart Casual to one person may be cargo shorts and a polo shirt to another dark blue jeans or kahki with a button down shirt or sweater.
My point is there are varying dress code definitions.