Another vote for Amtrak.

The drive will suck no matter what route is taken. By car, once you get to New Haven, the traffic will be unpredictable (well, predictably bad) all the way into Manhattan and 4hrs is the minimum travel time in good conditions. Actual travel time will be much longer and there's just no way to avoid it no matter the route.The coastal route by car, while somewhat nice and offers some stops, is best viewed from the train, comfortable and you can plan on when you'll arrive. If Acela is out of budget, take the NE Regional which runs more frequently and is still a pretty comfy ride. There's a bar/snack car onboard either train.