Middle_Seat, you'll have plenty of time for a couple of hours in Manhattan.

Worst case scenario, you get to Midtown at 3:30, tick around for a bit, and get back on the subway at 6:00pm which gives you 2.5 hours in Manhattan, and plenty of time to get out to JFK.

To address a previous poster, the fact that you're getting on a rush-hour 6pm subway out to the airport is mostly irrelevant.

It's a chance for an adventure of sorts - take advantage of it!