I have never been stopped in the one year plus that I have been living in Japan. I don't carry passport but make sure wife and I always have our military ids. Although not required for locals to carry id, I don't find it an onerous requirement for foreigners to carry ids. As for random stopping by police, I like OP, would be annoyed if this were a frequent occurrence.

Marc