The majority of U.S. citizens - I forget the percentage, but it's surprisingly high - don't even have a passport.

As of today U.S. citizens don't need a passport to re-enter from Canada, Mexico or the Caribbean, though it can be helpful in establishing that one is a citizen. That's considered a security hole and is supposed to change soon. (Whether the country one is going to will let a U.S. citizen in without one is a separate question, over which the U.S. has no control.)