As to where it would be less expensive to rent a car, picking a random weekend in September, renting on a Friday afternoon, returning Sunday Afternoon (for example).

To the OP - The numbers below are simply an example based on the last weekend in September. There's some miniscule chance that for the exact weekend you need the car they will not work out to your advantage to such an extent. You should most certainly try the different locations and see what you come up with for the dates you will need to rent the car based on your specific needs.

Renting from hertz in manhattan - $332 (it could be less expensive at the airports, depending on days and times)

Renting from hertz in new Haven, same times - $116, and $64 for train fare (for two), for a total of $180

Renting from Hertz in providence, same times - $67 (you need to pay an extra day, only the airport is open on Sundays to drop it off) plus $116 for Amtrak (for two) for a total of $183

Plus out of New Haven you will save an additional $15 or so in Gas, in Providence another $36 in gas over New York