Fatigues are popular casual wear in much of the U.S. I doubt there would be any problem if they don't have military insignia (unit patches, combat/award ribbons, badges of rank, etc.).

Of course, if they didn't carry a reasonably genuine-looking collection of that sort of stuff, they wouldn't get their wearer an upgrade either. I'm sure gate agents can tell the difference between a member of the military and someone who bought an outfit in an army surplus store.