I would bring attention to it on the way in next time. Perhaps there's something they do with them. If you don't want to do that, then I'd remove it.

The danger that I see in just leaving it in is that it might be noticed the next time you leave, and there's a tiny chance that they'd want to see the items you purchased without paying the consumption tax to verify that they were leaving the country with you. I assume that the position of the staple and hanko mark is such that they know for sure which entry the paper is for (otherwise, you could just remove it during your trip and claim the partial hanko was from a previous trip).