I wouldn’t worry about the situation as the airline can load in the passport details as required and do it differently for each direction of a given ticketed flight. Just make sure to have the tickets in the name of the passport being used for the first part of the trip, regardless of where you check-in. And for the return, check-in at the airport by first presenting the passport used for the first part of the trip but also having the other country’s passport on display. The airline reps at the airport should be able to do this without issue, but some of the less experienced reps may be sort of confused about what to do.

It’s very common for my dual-citizen relatives to have a different last name entry in their non-US passports than in their US passports, and it hasn’t been a show stopper for international trips even directly or indirectly between the countries of citizenship or for other sorts of international travel by scheduled air carriers.

At any government-operated border controls to enter and exit, present the passport that best facilitates/facilitated entry.