At most AA "customs recheck", it is just a end of a conveyor belt with a person with a scanner - to check your bag tag is OK or if they have to reprint. That person could not care one way or the other about collecting revenue for bags. I have never seen revenue collected there unless you are unlucky and had to get a new bag tag for example.
So I would suggest that you would be fine on your outbound - I cannot see QF collecting money for AA - but someone else may have experience?
On the inbound I would expect to pay, and not sure how they work that out as you should only be charged for the AA legs, but may become confused with the QF rules. It is usually though a case of the computer would say something.
Remember the US is quite flexible on the size and weight but not number of carry on bags. So you could also plan to pack a lot into that carryon - and even gate check it all the way back to AU for free. When they do that inevitable announcement for gate check bags.
So I would urge you though t have 2 bags, and then also more carryon than you might ordinarily want.
KF