OP here. I should have clarified. 90% of my travel with my child has been on American Eagle, so all elites are boarded first, and at the same time. The rest has been on mainline AA in F. We are not preboarding, we are simply boarding when the rest of the elites or cabin are called. And I do so for the normal reasons...overhead bin space, especially an issue when seated in the bulkhead row.
I can also promise you I'm faster at getting in and getting seated than most of the other passengers that I watch. So my question should have maybe been "all things being equal, is it inherently rude to board first?"
And yes, I've had more than one person tell me no one wants to be on a plane with a child longer than necessary, so we should be the last ones on.