This really varies greatly by airline. Excluding MPM fares, UA has fairly liberal rules for valid connections. AA has fairly strict rules for valid routing -- for east-west flights you can normally only connect one time, in either ORD or DFW.

As for why they do it, it's because they want your money. The airlines figure that, in the absense of some strange reason, no rational person would voluntarily chose to fly MHT-IAD-CLE-ORD-DEN-SAN-LAX instead of MHT-IAD-LAX. However, they might lose a passenger if they don't offer the option of multiple connection cities.

Assume that the fare on a given day for MHT-IAD-LAX is $650, while a cheaper booking class is available if you were to route MHT-IAD-DEN-LAX, which would bring the fare down to $330. Now, there's a CO fare in the market for $400. If UA didn't offer the longer routing, they would lose this passenger to CO. By offering the extra connection cities, they increase the chance that any given passenger will be able to find a cheaper flight.

If there were wide-spread abuse of the available routings, then the airlines would reevaluate them. But, they're relying on the fact that people generally just want to get from point A to point B as quickly as possible.

This is also the reason that one-stops are often cheaper than non-stops. Airlines know they can charge more for non-stops in popular markets because people are willing to pay more for the convenience of (a) less wasted time, and (b) fewer chances for something to go wrong. The airlines want to pick up that extra revenue. After all, airlines do more in the way of price-discrimination than any other industry...