You might also get a power of attorney in case of a medical emergency.
I have a feeling that the letter of consent would satisfy Japanese immigration officials, but you might face greater scrutiny when taking the child out of his/her home country. Not least because Japan is notorious for being uncooperative in cases where a Japanese parent has kidnapped a child to Japan in defiance of legal decisions and without the other parent's consent.
Is there anyone else with rights to access the child or shared custody? Could you prove that there isn't? I would suggest getting all documents notarized.
I am not a lawyer and none of this should be taken as legal advice.