The lounge agent was incorrect if it was the Admirals Club. If they were trying to access the Escape lounge (which is the one BA use in PHX) then the refusal was correct.

BA Silver gives you access to Flagship lounges and Admiral Clubs throughout American Airlines' network.