I just came back from a visit to Matsuyama castle in Takahashi (Not the same castle as in Matsuyama City.) and I did not see any other (obvious) foreigners. I exchanged Konnichiwas with most of the people while walking up and down so I assume it was mostly domestic Japanese tourists.
So, fortunately there are still "quiet" places to be found if you go off the beaten track.