But they don't always know the policy, and if they are wrong, it can be very inconvenient. Collecting the I94 is a good example (and I may be wrong, but I interpret the comment about going to secondary to mean that the OP needed to get a new I94, not that s/he has a flag on his/her file). Lots of AC agents think they are supposed to collect I94s from all outbound passengers who have them. This is simply incorrect. So "simply produce what you are asked to produce and leave it alone" is very unhelpful advice.
Often1, my problem with the gate agents is that they are the last word (the way only front line bureaucrats can be) when it comes to a issue when that agent is patently in error. And that error will cost me tangible wasted time / cost.
SYM is right, I meant that I had to go to secondary on the next CAN->US trip to get a new I94. And the CBP agent told me that the agent at DCA was flat out wrong. (And that piece actually had a "Multiple Entry" stamp -- but not "do not remove"-- on it, yet the DCA agent still took it.) That additional I94 visit took an hour of my life and I was lectured by the CBP about the DCA' gate agent's mistake.
And the SFO agent this time tried to force me check a carry on that actually fits in the metal grill 20 min prior to commencing boarding (so no excuse about bins-full). If I hadn't objected, I would have to wait ~X min for the carry on to come through baggage claim. (I say "X", because those things vary widely based on luck)
I thank the other posters for their experiences though.^ From now on, I will start re-printing boarding pass at airport wherever I can. Hopefully it will avoid future unpleasantness.