I visited a friend once who had a house in the Hamptons. I took the Hampton Jitney both ways. The majority of the rest of the customers on the bus seemed to be household help for people summering in the Hamptons.
Once there, my friend drove us around. I tend to think it would have been pretty boring without a car based on that experience, although it also was not that exciting even with the car -- a few decent views, but mostly like being in Manhattan with more trees.