When taking the Airtrain to Jamaica, you purchase the $5 after the trip in Jamaica. You'll see the ticket machines right before the turnstyle. I may be wrong, but I don't think you can purchase the LIRR ticket from those machines but just Metrocards themselves. Once you pass the turnstyle and go the LIRR (just turn left and enter the LIRR upper platform), you'll see LIRR ticket machines.

I do know that the LIRR ticket machines will also sell you a metrocard but I don't think the reverse is the case.