Life is much, much easier in Japan when renting a car if you have the IDP. It really is.
That being said, I have on one occasion when I did not have one convinced an agency to rent a car, and that's without speaking or understanding Japanese. So I can verify it can be done but, as others have said, I wouldn't count on it.
The IDP is not much of a document. If you happen to have one that has expired, just put a sticker over the date and write in a new one. That's worked every time for me; i.e., more than once (but not a statistically significant number of times).