Can't say anything about diving, but your "requirement" of a quiet island excludes Mykonos, Santorini, Ios, Spetsai, Hydra (yes!), certain locations on Crete, Corfu, Kos, Rhodes and Zakynthos and several more. Based on the need to get there by plane or "easy" ferryboat excludes, for example, Samothraki, Paxoi, most of the Cyclades and Dodecanese. You are still left with a lot of choices, though: Cephalonia, Naxos, Syros, Tinos just to list a few of those I've visited.

Cave: I'm certain there will be answers coming up where people will be mentioning the one island they visited, which left them with great feelings. The opposite is also true, I for example hated an island that is high on the list of many people (won't say which one).

Conclusion: there is a very, very high probability that you'll have a great vacation whichever island you choose!