You are correct. The OP will need to see an agent at check-in to make sure that UA have details of both passports. All airline systems have the ability to record 2 passports against a check-in - one for the departing country, the other for the arriving country.

Australia carries out electronic checks for visas/ETA at check-in - attempting to check-in with only the US passport will fail (due to no visa). Attempting to checkin with only the European passport should work, but will both not result in Pre-check being given(*), and will not record the exit from the US correctly. On arrival in Australia, use the European passport at the "Smartgates" to pass through immigration.

On return it's a little different as Australia has physical outgoing immigration. So you can use the US passport to checkin (so as to be given approval to check-in by the US government), and then the European passport at Australian exit passport control.

Or just fork out the US$14 to get an eTA on the US passport, and use that one everywhere.

(* Pre-check might actually be given, based on the grounds that some foreigners still receive "random" Pre-check - but it's far from guaranteed)