This has nothing to do with UA and really belongs in the Japan thread. OP should thank UA for having called. Most carriers would not have caught this until the document check at departure or, if they did, not have called.

As others note, transit has always involved three separate jurisdictions: origin, transit & destination. Here, even if UA researched and saw the return ticket, this would not be a transit.

As others note, OP will need to find a third nation which will permit him to enter and then fly back from there. Even a round-trip to NRT won't work because there is no transit.