The New Otani's pool is open to the public, outdoors, centrally located, has great poolside dining and sometimes live music, but it's pretty pricey (12,000 yen during the week, more on the weekend, cheaper if you go after 6 PM for the "night pool").

http://www.newotani.co.jp/en/tokyo/i.../index.html?mv

The Toshimaen and Yomiuri Land pools are going to be chock full of little kids no matter when you go.

If you're going up to Toshimaen you might want to check out "Niwa no yu," an onsen on site which has a mixed-gender pool in the middle (bathing suit required, natch). It's indoors but should be a more relaxed environment.

http://www.niwanoyu.jp/floormap/index.html