I haven't been to Naoshima and hadn't understood your bias. Perhaps Takayama would be better served on another trip - next time, stay longer and combine with visits to other towns and villages in the region.

Just in case you're tempted, I'd advise against the Hakone Open Air Sculpture Museum - IMO: Modern Art for the kind of people who like Monet tea towels. Big disappointment.

For architecture, I suggest you don't overlook Osaka (Namba area has some great buildings).

On another future visit, these places may also be of interest: http://www.sapporo-park.or.jp/moere/ and http://www.miho.or.jp/english/

And you might get a lot out of http://www.meijimura.com/english/index.html in Inuyama. It does a beautiful job of showing how Japan went from the kind of buildings you see on Nawate Street/Kurashiki or at the Nihon Minka En to what you'll see here:http://www.naoshima-is.co.jp/english/first.html