My experience has been the same. The really rude masses of tourists tend to be foreign visitors, not Japanese. One of my pastimes when in crowded areas, like Nakamise-Dori in Asakusa or Kiyomizu-dera or Heian Jingu in Kyoto is to see how many languages I can identify. In crowds, most have been something other than Japanese.