When the airline checks your passport at check-in, it's to make sure you're eligible to enter the destination country. I believe there are fines to the airline if they put you on a plane without a valid passport or visa for entry into the destination. For that reason, dual citizens often check in with the passport of their destination country (when I go to Italy this fall, for instance, I will check in with my Italian passport at BOS. When I return, I will check in using my US passport at FLR, even though I will show my Italian passport to FLR security and CDG exit immigration).
The law is generally that you must present your local passport to local authorities of your countrie(s) of citizenship. The airline checkin agent is not a local authority, and the US doesn't have exit controls - they can implement them if they want to, but they haven't. There might be some data feed from the airline to DHS, but that isn't really your problem to solve for them. When you return to the US, you are required to show your US passport to Immigration in the US. An expired passport might make things a little bit difficult, but an expired passport doesn't equal expired citizenship and you still have a right to return home. You might be hassled about it, though, so I'd get the passport renewed.