I have experienced this directly on AA's website as well, not just Google Flights links. I am under the impression that "married segment" logic may be the problem, but I don't really understand this so can't comment. I've experienced this multiple times over the past 8 months over several days in each case; the outbound/inbound search in AA.com will present certain flights as options but will not allow booking in certain combinations. Since they keep coming up over days, I don't think it's an issue of someone grabbing the last available seat, or a temporary mismatch in available inventory. Rather I believe some kind of validation process at the end is using different criteria than the initial availability search, causing the problem.
For me, the only flights that it affected involved certain combinations of widebody domestic, e.g. A330 SFO-PHL-XXX with XXX-CLT-SFO on the return, so I was wondering if that was part of it. (I couldn't get the widebody both ways).