I think the OP is correct to pursue BA Holidays. The OP didn't form a contract with AA. AA are a contractor to BA Holidays in this circumstance.

The OP paid BA Holidays and formed the contract with them for travel which included baggage and the baggage allowance was shown on their e-ticket. So following the money, it is for BA Holidays to refund their customer for the incorrectly levied luggage charges at check-in, and for BA Holidays to subsequently sort this out with AA as their subcontractor.