I always fly casual unless its a really short flight and/or I have to go immediately to my appointment.
Reason: I change to business attire when I arrive so I have a "fresher" appearance than I would if my clothes were getting wrinkled, etc. on the plane.