Stopped once in 11 years for a passport check, obviously didn't have it or my gaijin card either. They were happy with my driver's license. I routinely never carry ID here. The Japanese don't have to, why should I?
To the OP, you're just unlucky.