OP has a pretty good problem to have tbh. They are treating you as Japanese while a lot of "more Japanese" foreigners will never get that kind of acceptance.
Anyway I can kind of understand the self consciousness. You look Japanese but can't do some of the most basic things expected of you. So people sometimes look at you like "... is wrong with this guy" which probably wouldn't happen if you had blond hair and blue eyes (just an example). I have family in China and Japan and I get that feeling as well in both countries.