I would never skip Ha Long Bay. It's been a few years since I've been, so I admit it might have changed and be more of a hassle in some ways, but it's one of the most beautiful and interesting places in Vietnam, imho.

I'd definitely go there and don't think your schedule is bad at all. 2 nights in Hanoi, then take an early morning trip to Ha Long, and boat then and overnight there or on the boat. Then fly the next day to Saigon, you have that half day in Saigon, the next day also or else could do a day trip to the Mekong. Another night in Saigon. Works fine.