I have repeatedly been through Incheon with dogs on a leash, and there never was a problem.
As there are no pet relief areas in the post-security transit area, you will need to leave the airport anyway (and enter Korea, meaning your dog needs to comply with the regulations including rabies titer blood test etc.).
For an eleven-hour layover, get a (day) room in a pet-friendly hotel (I can recommend
http://www.incheon-hotel.com/en/day_use.html - inexpensive & convenient). and take doggie on walks around the neighborhood. There's parks and forested areas near the hotel I linked.