It really depends on the rules of the airport and whether those rules are enforced. When I flew with my Yorkie through STL and LAS, he was rarely in his carrier. In LAS, he walked on leash even on the tram to the D gates and I would carry him in the fast food restaurants (I always asked and never took him into a restaurant) and was never questioned. STL was never a problem. EWR was always a problem. I had some air crew literally yell at me and another couple with a dog on leash to put the dogs in the carriers. They weren't just enforcing rules (what exactly does air crew have to do with airport regulations?), they were extremely nasty about it. I always sat with him on leash in a waiting area away from people and no one said anything about that and most people were just wanting to pet him.
I checked the website
http://www.ifly.com which lists all major airports and it appears that all the airports I checked required pets in carriers so it really depends on the enforcement.
Of course, none of these rules applies to service dogs.