1. Foreigners are required to carry a gaijin card or passport at all times.
2. That said, I'm not sure police have a right to randomly check passports.
3. I've been asked for a passport a couple of times and when I asked if it was "gimu" (mandatory) that I show the passport, they said no.
4. I've confronted policemen who were hassling a Filipino woman I happened to know for her passport. When challenged they quickly backed off.
5. Japanese police can not ask Japanese for a passport. So unless they have proof that you are not Japanese they can't require you to produce a passport, and you have no obligation to answer any questions including "Are you Japanese?"

Unless you have time to waste or are interested in making a point about human rights, police excesses, etc., it's usually easiest to just carry the passport and show it.