Yes you can book the AA part separately, with cash or miles or cash+miles, but it will end up on a separate ticket so make sure you have adequate connection time and insurance.

In most cases BA would through-check your bags on to your second flight. But US immigration requires all passengers to collect their bags and take these through immigration. You'll need ESTA too.