Good question. I'm conflicted. When I arrive to the gate after priority has boarded there is usually a long general boarding line. The priority line is usually slightly off to the side of where the agent is standing, and you often stand there without getting noticed. I've stood in the priority lane for a minute or so without the agent noticing and motioning me forward. At that point you have to step over to the front of the general boarding line. When you step in front of the person who is next on the GB line it sometimes generates nasty comments, and always generates some nasty stares from a number of people in the line, as many are not familiar with priority boarding, which is why Group 5 is often blocking the gate when only F has been called.
There have been enough times when I have gone to the front of the GB line by cutting in front through the priority line where I received negative comments or stares that I usually just go to the back of the GB line. Or, I go to the priority line and wait to see if the agent notices and waves me forward. Usually they don't, and it's not worth it to me to say, "excuse me," and step in front of the next general boarding passenger, so I go to the end of the GB line to avoid the friction.