I would vote for Boston, though I freely admit that I am a biased since I went to college there. A lot of Boston is walkable, and quite compact, the public transport there is excellent too, allowing you to make most use of time. If the weather isn't so good it won't matter, there's plenty of ways of staying out of it.

If you wanted another ideas: Montréal would be on my list too, especially from a food point of view. Having said that, there is an inevitability with NYC, not least in terms of price availability.