I go for this approach: when I put the earphones/headphones in/on, hold them a couple of inches from my ears. If I can hear it, I turn it down until I can't.
Occasionally, though, people do ask you to turn the volume down, although what they're hearing is actually somebody further away who has theirs turned up even further!