I visited both a few years ago and loved both of them, beautiful scenery, amazing food and people.
Ishigakijima has probably a bit more to do but if you're the "luxury" type then neither are ideal.
Despite the beauty, as often in rural Japan, some things are a bit run-down and thankfully (for me) I didn't see big international hotels.

Couldn't imagine not having a car though, might be worth trying to find a way to get the translation of your license, pain as it may be.