Definitely worth seeking out festivals--they add irreplaceable local color. So do weddings (ie, weekends at Meiji Shrine) and antique markets (various times throughout major cities).

A JR rail pass for at least a week is highly recommended...especially if you like temples and shrines. Kyoto is a must for that, and I'd add Nara and Arashiyama, too. Oh, and can't miss: Miyajima and Himeiji, both doable as day trips from Kyoto.

Since the JR pass is good for travel in Tokyo on JR lines, and for use on the Express from the airport to Tokyo, the 2 week one is probably a good choice for all of you; then, you could consider doing JR East for further exploration from Tokyo. And, of course, digging deep into Tokyo and its environs.