The ATS is outside security, so when you arrive at T3 you will need to go back through security there. There is no way to move from T5 to T3 inside the secure zone. There is also no practical way to get from T5 to T3 other than by taking the ATS. I suppose one could walk, but it's a fair distance and not designed for pedestrians.
Watch carefully as you leave the Immigration/Customs area. The process is:
stand in line to clear immigration
wait by the carousel for your bags
walk through customs with your bags - you will either be directed straight through or directed to an inspection line
Once you pass the customs station you will go down a short hallway WITH YOUR BAGS, and there will be a place there where AA will take them from you. If you pass all the way out into the terminal you will need to take them with you on the ATS and then recheck them when you arrive at T3.
The good news is that the route is pretty will signposted and my advice is to just throw away the map and follow the signs. When you come out of immigration/customs you'll be on the lower level of T5 and there will be an escalator up almost immediately in front of you. That's where you want to be.
There is no T4, so T3 is the first stop on the ATS. The only tricky bit is that the ATS serves T1-T2-T3-T5-Remote Parking. If you don't pay attention at T5 you could wind up on the train to Remote Parking instead of T3. Even this isn't a disaster since the trip takes like 2 minutes and you can just stay on the same train for the return trip.