Well, with just under 4 full days it stands to reason hopping to another island won't leave you with as much time. Figure a few hours' time on one day just making the trip to the other island (factoring in getting to the airport, rental cars at both ends, wait time, checking in and out of hotels, etc). But it's your vacation so only you can decide.

One can certainly spend more than 4 days and not even begin to appreciate Kauai. Is it enough time for you? Don't know. What do you want out of the trip? What sorts of things do you and your family like to do??...active vacation, beach bumming, nature, cultural, historical sights, fine dining, etc, etc??