I've made multiple trips with my children, who have a different last name listed on their passports used to enter Japan. They also use a different passport from mine. Just me and the kids, spouse is not present. We've done this a couple of times since they were born so multiple occasions when they were < 12 years old.
To an observer, my situation might look the same as yours. Unlike your situation, we do not need visas.
Nobody has asked or raised any questions entering or leaving Japan, although admittedly if they did, I am the biological parent and we do have other passports that establish this relationship.