Depending on how long you stay, the fantastic train system, not just the Shinkansen "bullet" trains but the regular trains as well, gives you easy access to rural areas from both Tokyo and Kyoto.

On a previous, I did a long day trip from Tokyo to the area around Sendai, half way to the northern tip of the island of Honshu. A day trip to Nagano, site of the 1998 Winter Olympics is also perfectly feasible.

From Nankai-Namba station in Osaka, there is a direct train to Mount Koya, site of dozens of Buddhist temples that take overnight guests. It's a scenic trip, once you're out of the urban sprawl.

Remember that Japan is only about as big as California.

Get your guidebook and look at the "day trips from Tokyo/Kyoto" sections.