In general, will TSA allow you into a terminal so that you can access a luxury lounge (Centurion, Priority Pass or any other lounges), regardless of whether you have a boarding pass for an airline that is serviced by that terminal?

I am flying out of San Diego, and the Airspace Lounge is in Terminal 2. I am flying Southwest, which is serviced out of Terminal 1.

I am asking for a specific scenario in San Diego, but my question is really a general one, since I imagine this situation happening often. Example: The Centurion Lounge at SFO is in the United Terminal. But often, I am not flying United, but want access to the Centurion Lounge.

Thanks!