What is your basis for making that statement? The TSA has the right to search at an administrative checkpoint. Two guys walking up to you are not an administrative checkpoint. It's just a consensual encounter. The TSA can't arrest you or detain you or eject you; they are not police officers. As for the police, I also believe that the police have no right to eject you if you refused to consent to a random search inside that area. You have just as much right to stand there as all the other passengers, including arriving passengers; using the restaurants, shops, etc. What would be the justification?
Now, the TSA could stand at the gate and probably order the airline not to let you board. What would the police say if you then decided to do some shopping (like many of the other people there?). Perhaps the airport manager could demand that you leave, and invoke a trespassing charge if you do not.