I'm assuming on any given days gate agents will deal with 100 people who they think are in the wrong line. 98 are going to be in the wrong line. The 2 that aren't are the people posting on threads like this where they feel "discriminated" against.
The vast majority of 1st class passengers look a certain way. They're usually older than 25, they usually are dressed well, and so on.
So when a 22 year old wearing flip flops and a and a hipster ironic t-shirt shows up to the counter, it's a very reasonable for the GA to assume this person is not in the right line. They're not mean people, they're not out to get you, they're just making an educated guess about which line you should be in. Don't get so defensive.