The only tepid baths I can think of off hand are in places that are typically labeled "Kuhrhaus". They tend to be a compromise between a swimming pool and an onsen. Co-ed bathing in a warm pool with swimming costumes on... Not entirely unpleasant. Generally there are jacuzzi's and other things around the main pool. And they have lots of aquarobics classes and so on.

Having said that, I don't know of any such facility around Yoyogi...

I suppose there is always the olympic swimming pool, which offers a very large, chlorinated, pool (at bracing temperatures) and plenty of showers...