FWIW, I have had some coworkers who cringe when I mention that I'm going to Europe by myself. Some fear terrorism, some fear general anti-American behavior, varying levels of crime, and others think that the film Hostel is real.
Despite the fears of others, I suspect that for most, it's either family responsibilities, a lack of money, or little to no paid vacation time that hobbles their ability to travel.