Unfortunately, I agree that the responsibility for obtaining the necessary travel documents belongs to the traveler. The only caveat I think that might make your brother's travel agent responsible is if they directly advised which visas were required (i.e. if they told him he needed a visa for Canada but didn't tell him about transit visas for the US).

You might want to try to speak with the airline directly about getting a refund; it's not likely they will refund a non-refunable ticket, but it's worth trying.