You're not missing a lot by skipping the Tokyo tour. We were essentially guided along the driveways to view the buildings from the outside. Some architecture and landscaping/trees were a bit interesting but overall, it's a skippable activity.

While I also wouldn't consider it must see, I enjoyed the Kyoto tour as the architecture and grounds were more interesting. As emma69 mentions, you can look into some of the buildings which we were not able to do in Tokyo.