There is one problem with your plan.
Although I agree in principal with your intended approach, you should still follow RichardInSF's suggestion of local guides and walking tours which you can venture from at will.
The reason is that Tokyo diesn't share the same architectural "vocabulary" as other places you've been to and you won'5 be able to pick up on written signs or cues either.
The problem is that you often won't be able to pick up on interesting discoveries just by passing by them.
I understand that visiting Tokyo demands a Hackett Knife. Just start the process before you arrive; settle on neighbourhoods, but then limit your research to those neighbourhoods, even if it's just a couple of articles or guides, and note down a few notable addresses within that area solely so that you don't just walk past them.
For you, I'd suggest the YaNeSen (Yanaka,Nezu, Sendagi) area with the added bonus that in May the azalea festival at Nezu shrine is in full swing (more crowded on weekends, but also more live events).