You should be able to enter the US with an ESTA on your EU passport. Not sure about re-entering Canada though. The Nexus card should technically be sufficient at the kiosk, but I don't know how CBSA would feel about you showing up without a Canadian passport.