Initially, the main reason for permitting only ticketed passengers through the security checkpoint was to alleviate congestion---fewer people in the screening lines would speed up wait times.

Nowadays, with secondary screening performed at the main checkpoint instead of at the gate, all passengers must have a boarding pass to get through the metal detectors (an itinerary or paper ticket won't do) so that TSA can check for the scarlet letters of SSSS on everyone's BP and single out everyone who is subject to extra scrutiny.

Under current TSA policy, if non-ticketed passengers were to be allowed airside once again, they would all have to undergo secondary screening.

For now the refundable ticket trick is pretty much the only solution for getting through security if you're not flying.