Who’s they? The priority security line is managed by an e-gate not a person. Maybe you were at the Nexus line?
edit: unless you were transborder? I’m thinking of the domestic/intl gates.
Yes it is a e gate and initially I was in the queue and there was a person scanning the boarding cards and when I saw the * G , I asked him if I could go there he said no. Stay here.