6 pm is a fairly busy time as there are still some European flights arriving but you should be ok if you are able to use the US immigration lines.

If you have to the use the lines on the other side of the hall - I would be worried. The process of the photo and print taking slows things down.

Once you clear Customs & Immigration, and re-check your bags for the connection, you will proceed upstairs to catch a mono-rail type train that will take you from terminal 5 to terminal 3. The ride takes about 3 minutes, and I think they run every 5 minutes or so.

The TSA security lines will be about 20 minutes. I will seem like there are a lot of people, but the lines seem to move fairly quickly.

There is another security line at the end of terminal 3 - the FAR right end of the terminal that puts you out by the G concourse. There was a time when this was little used and you could count on short lines. Sadly AA has moved some check-in closer to here, so more people use it now.

You will have plenty of time on the domestic->international to make your connection. You will NOT have to clear security again.