As others have said, just smile and nod as you walk past them. There is no reason to be rude, and I do try to humor them by showing my boarding pass, but they don't have any authority to physically stop you (as far as I know).

Not quite the same issue, but I have had them aggressively yell at me that I need to go into the Skypriority lane (as opposed to the general boarding lane) and that somehow I can't go down the general boarding lane (I literally had one chase me down and tap me on my shoulder and say I was going down the wrong lane). If there is no line, I prefer to go down the general boarding line (as opposed to Skypriority) because it is actually shorter than the "L" you have to make in the SkyPriority lane and it avoids the hawking from the CLEAR agents / sales reps when you walk by. But according to them I "have" to go down the Skypriority lane because, you know, reasons.

I don't fault them personally. I am sure they receive dirt poor training and are just trying to do what they are told to do.