I used to carry mine on me at all times, but have since stopped and usually leave it in the hotel safe. I have a US and Irish passport, so if in a country that isn't as stable/safe as say Europe, I'll carry a copy of the Irish one, which is typically what I use to enter/exit most countries. Always either in front pocket or side cargo pocket (which had snaps and a zipper).

I also have scanned copies sent to an email account and all pertinent info, such as number, issue date, and expiration date memorized.