What abmj-jr says.
Still, flying to and from Shikoku shouldn't be very different to getting the Shinkansen to Kanazawa. It's the crazy schedule you've set yourself, Shikoku and Okayama both benefit by being explored at a gentler pace. You've already seen many of the sights; give yourself some time to soak up the atmosphere.
And yes, Naoshima is wonderful.