So I asked how come the border agent (upon reentering the country) said my passport was fine but the AA agent said it couldn't be used for check in.

I guess I won't be flying AA international anymore until I get my new passport ...

Response is below.

"Since we are subject to substantial government fines for transporting customers without proper documentation, our agents utilize a sophisticated documentation data base to verify requirements. In addition to ensuring the correct documentation, they also must make decisions about whether or not to accept customers for travel based on the condition of the documentation in their possession at the time of departure. The issue of damaged or mutilated documents has recently become increasingly problematic for us because passengers are being deported on this basis. As such, an alert was sent to our airport managers about this issue. I'd also like to point out that it's discretionary both for our employees and Immigration officials. What is considered acceptable by officials here in the United States has no bearing on the response at the passenger's destination. Therefore, we expect our agents to err on the side of caution. We do understand that you were advised otherwise by other officials but, given the possibility of heavy fines, we simply cannot make exceptions.

While we are sympathetic to what happened, we must rely on our airport personnel to evaluate these situations and act accordingly. It is not our policy to second guess such decisions, as we have not physically examined the documents. No doubt my response is disappointing but I hope you can understand our reluctance to take any risk that may result in our being fined. Further, such matters are completely outside of our control once a passenger is onboard and if deported due to an unacceptable document, the cost of returning the passenger is the responsibility of the airline. With that in mind, our position remains that documentation must be in very good condition for us to allow travel. Still, we've shared your perspective with our Station Manager at San Francisco.
"