When is this itinerary for? Winter, Summer, Spring?

With a 2.5year old I agree about a “slower” pace, not to make the visit actually slower but to ensure play time, play time and more play time.

My number one recommendation when visiting with a kid that age is Akebono Kodomo No Mori on the outskirts of Tokyo. Free to enter. Pick up your own picnic lunch en route as food isn’t sold there. (Cafe Puisto with organic food and an integrated playroom allowing parents to drink coffee in peace opened recently, looks delightful!) Leave the Ghibli Museum for when the kid is older.
Other toddler highlights are:
The Nihon Minka En (just having the space to play peacefully with sticks and stones and discover nooks and corners in this environment is magical.)
Showa Kinen Koen - a Park as fabulous for adults as it is for children
The Benzaiten cave shrine in Hase Dera Temple (as an adult, you’ll need a good strong back for this, the ceiling is extremely low)

Both children and adults’ expectations for these children differ as wildly as with anything else. My own was exceptionally active and, let’s say, “unpassive”. She rarely slept during the day. Her surprising appetite for good sushi at 2 and a half (her Japanese grandfather was doting and generous in this regard) is still remembered as family legend and chuckled about, to this day she remains hugely fond of ikura-salmon eggs.