The Flybus is a coach to the bus terminal, then a van to the hotel. You don't need to book in advance unless you really want to as I believe it's the same price, nor do you need to specify a time on the outbound trip, as they always have buses available to meet incoming flights. If you will definitely be using them to get back to the airport then buy a return as it's cheaper. You can pay in the airport or on the bus, using card, ISK cash or even a combination of both. There are no trains in Iceland.

You could also take Gray Line which appears to be cheaper, but I believe it takes longer as its terminal is far from the city center.