No need to feel guilty. You have priority. You would have boarded before the other folks if you were early, so you're not "delaying" or "inconveniencing" anyone, just exercising your pecking order. When they board Group 3, they're also boarding anyone remaining from Groups 1 and 2.

That said, no reason to be obnoxious or snotty -- get into the Group 1 lane, don't elbow your way, catch the eye of the GA, who will wave you in.
If they are boarding Group 3 and the line is short, you can hop into it just for convenience, but if it's long, exercise your privilege but noblesse oblige.