This might not totally apply to your situation. I had similar type of trip to Japan with one of my good friends. He's a tall, white American guy while I can pass off as Japanese. I think the only situation where his appearance came into play was when we were stopped by the local police for an ID check. In my perspective, it may be because he was definitely a foreigner and the Japanese are pretty strict with immigration. Other than that, we had no problems in our trip. I would just be prepared and make sure both of you have your passports handy. They asked for mine when they realized I could not understand a word of Japanese.