Yes, I do not step outside the door in Japan without my passport when I'm visiting. Even long-term expats will not leave the house without their alien registration.

I know people who have gone out without their passport or alien registration (as in "Why should I have to take my alien registration along on my morning run?") and encountered a cop who is either bored or in a bad mood and asks for their ID. From what I recall of their accounts, the cop has the right to haul you in, make you pay a fine, and, in a typically Japanese touch, write a letter of apology.

Your passport is unlikely to get stolen in Japan, and it's no trouble to stick it into a pocket or neck pouch or money belt or handbag and keep it with you at all times.