Woah, sorry you had a bad time there, I couldn't have had a more dissimilar experience. I actually went in the winter time, perhaps that is why there were no crowds then. Saw few boats. I went with something mid range, didn't do luxury because it was a last minute whim. Turned out to be the best decision, free upgrade and the cost was super low for 3 days 2 nights (~220$ for two people).

Roads were also paved by the time I visited, which was 2 years ago (December). I will say that it is definitely worth hiring a driver with a comfortable car to get there and back.

As for weather ---- it was cold but the bay was strikingly beautiful all the same. Air was crisp, and I still ended up jumping into the water on a sunny afternoon. I didn't expect to enjoy it, was hesitant about going because of the cold but it did end up being the highlight of my trip along with Angkor Wat.

3 days though, no matter what is too short as people have mentioned. Maybe keep it in mind and go back another time, definitely worth seeing.