I think the answer depends quite a bit on how you access the airports, i.e. if you take car services/taxis vs. public transit. I take public transit almost exclusively to NYC-area airports and on that alone I prefer JFK -- the ability to entirely avoid road-based transit for me makes travel time a lot more reliable. Of course, I realize the subway isn't perfect, but as of late I've started taking the LIRR to cut out the subway trip, as I live relatively close to Atlantic Terminal in Brooklyn (many people take it from Penn Station of course).

If you're taking a car, I think the calculus is different but as I do it so infrequently I feel I can't really speak to that.