The UK does not need evidence of an onward/return ticket, but if the OPs family was refused by Italy, it's unlikely the UK would let them back in again with no return ticket to the US, at which point they would need to be repatriated by either BA or AA, so caution from AA is a good idea even if they were not going to Italy that night (but there is still no excuse for rudeness). My USC OH enters the UK on the return portion of a ticket every 6-8 weeks, and has been asked for her onward ticket on at least two occasions that she's mentioned to me this year. Thankfully she had it printed and available both times but if she didn't I expect a lot more questioning would have been in order.

On the other hand, with regard to AA not warning you earlier about their requirements for boarding a flight to a a specific destination, I really don't see why they can't display this at the time of booking. Someones suggestion that AA would have to call everyone and add 20% to the ticket price is ludicrous, but just to have it listed with an option for linked ticket number or visa would make sense. It would also save check in agent's time at the airport. I am currently in the US and entered on a OW award ticket, with a separate PNR for my return, I was asked on no less than 3 occasions at the airport to show my return ticket (AA Security, Check in and gate) - if I could have entered my other ticket number online to show my return ticket, it would have saved a lot of time for me and them. Yes, it's not their responsibility to tell us the rules, but they are going to tell us by the time we get to the gate if they think we may not be following them anyway, so why not do it at the time of booking. It's a bit like booking a table at a restaurant, only to be told when you arrive that they have a table for you but you can't come in because you don't meet their dress policy and you won't be coming in until you get changed because the manager may later throw you out if he sees you - AApologists would probably say "well you should have looked it up online or asked when you called to book", but any reasonable person would think they would inform you at the time of booking.