I only carry my passport domestically if I think I might need it for a change of plans; but typically that's rare that I fly international last second, so usually my wife could always fedex me the passport if it came up.
I don't like to carry everything all the time in case something gets stolen. It's the same reason I always keep a credit card in the hotel safe with my passport -- should something happen while I am out, I'm not at the mercy of getting a replacement card.
Steve