Even though others have reported not being required to show it, the IDP is required to be accompanied by the underlying license in order to make it valid. Let's say you have a US driver license, an IDP, live in Japan, and have a Japanese driver license. Why would you carry IDP, US license, and Japanese license? Well, if you get a ticket, wouldn't it be nice to have that written against the US license and not show up on your Japanese driver record? And this is assuming the officer doesn't decide it's too much effort and just lets you go.

But this gets into sketchy territory. If you have been in Japan more than a year, then the IDP is not a valid license to drive in Japan. Depending on the thoroughness of the officer, he may or may not figure this out (such as if he asks for your passport or ARC and figures out you've been there more than a year). At that point, he might try to get you for driving without a license, at which point of course you'd produce your Japanese license. That would get you out of the driving without a license charge, but you'd also probably have one pissed off officer, as it would be obvious from the chain of events what you were trying to do, and I doubt that feigning ignorance or a language barrier would overcome that perception.