I have decided to stay downtown after all. Agree with FTers here that it’s better to stay in the location you want and accept the extra 10/15 mins on the subway. I’d say exactly the same if someone asked about London!
Have now booked The Maritime Hotel, next to Chelsea Market.