It all depends on what you like.

Personally, I hate packing and unpacking. If I base myself in a central city, like Tokyo or Kyoto, the marvelous Japanese train network ensures that there's more to see within day trip distance than I can possibly cover in one trip.

For example, during one stay in Tokyo, I made easy day trips to Nagano, Mito, Aizu-Wakamatsu, and a client in Fuji-shi, and spent a night with some Japanese friends who have retired to the Nasu area.

From Kyoto, I've ventured to Nara, Osaka, and Kobe (of course) but also to the burial mound area, Yoshino, Tenri, Uji, Muryoji, and Hikone.

On the other hand, sometimes rural peace and quiet is what you need. I once spent a night in Ohara just for the atmosphere, even though the village is on the Kyoto city bus line. For Koya-san, a night in a temple is an integral part of the experience. In other cases, such as when traveling in more remote areas, you have to move from place to place.