You can get to Ostia Antica by public transport from FCO. Just take a bus (not sure what frequency) towards Ostia Lido, and cross over the highway to get to a rail stop (the line that runs between Roma Piramide and Ostia Lido) and go towards Rome for one stop, and cross the highway to get to Ostia Antica. Only trouble with this I don't think you should/can rely on the timetables for the train and/or the bus.

There isn't much around the place (it's surrounded by a very suburban area) where you are likely to find a taxi so you may end up stranded though so close to the airport.