This ID need not be a passport. The default ID I keep on me at all times is my drivers license, hey it even has my blood type on it if we're talking medical emergencies. It's a lot less of a hassle to replace compared to a passport, and it's not always about being afraid of pickpockets, rather just misplacing when out and about.

That said I usually bring my passport out with me when I'm in Tokyo - because tax-free shopping