We stopped renting a car on our Hawaii trips some years ago after we realized the car mostly sat in a parking lot and just got used once or twice in our 8-10 day stay. On more recent trips, we’ve just rented a car for a day or two as needed for any sightseeing trips where public transportation is not a convenient option.

so I don’t think your plan is crazy at all. Worst case, you can always try things out and rent a car mid-trip if things aren’t working out as well without one.