Welcome to Flyertalk and the BAEC forum sro1954.
Did you purchase the ticket from BA or AA? If the former, then the agent was correct that you could not upgrade with AA miles. You would have had to book with AA in order to do that. Regarding the cancellation - the agent was correct to advise you that you needed to book with AA (if you purchased the ticket from BA). Did they specifically advise you that you would receive full refund or did they just advise you to book with AA? Judging by the amount you paid for the flight you probably booked a non-refundable fare so you were only due governmental/airport taxes back. Did you enquire what the refund would be? If you did and was promised a full refund then it would appear that you're owed one, but if you canceled without finding out first what you'd get back then it appears to be a very costly mistake unfortunately.