Nationality Law is pretty clear on holding dual nationality after age 22 and when discovered they can force you to give up the other one if they want.
How would they "force you to give up the other one"? They ask you to declare to the Japanese government that you want to keep Japanese nationality. You have a
douryoku gimu to complete the procedures in the other country to give up your other nationality, but there is no penalty if you fail to do so.