Seems the issue is with ANA?
Why couldn't the luggage go back to carry on BTW?
Otherwise, why not paying and then arguing with AC/Aeroplan later?
This said, it seems very possible that the rules actually were in ANA's favor. But then we get into the more complicated issue of which rules apply in this sort of connections and what if the airlines which checked the luggage did in realtion to these rules. Anyway, issue itself does not sound very clear.