You didn't need to do the ESTA or anything with the Japanese passport. An expired US passport is acceptable by CBP. All of the stuff about a US national using a foreign passport with ESTA is literally for a US national who does not have a US passport. Your kid did have a passport, it had simply expired.

US citizens don't need a passport at all - valid or expired - to re-enter the US. You just have to have your identity and citizenship verified, which will lead to a delay at re-entry. US citizens cannot be denied return entry to the United States. Your biggest issue would be the airline not letting you on the plane, or something of that sort.