*Generally* when gate agents make checking bags compulsory due to lack of overhead bin space, they’re doing rollerboards and or large duffel bags. I’d be surprised if they made you check a messenger bag. What are the dimensions of the bag?
just as an aside if you get one of American’s eligible cobranded credit cards you’ll get priority boarding on every AA reservation, Which would make it much less likely that you would have to check your bag.