I was at the Poly for a week in June and didn't have any problems. But I also wasn't expecting much. Heck, I've learned never to expect more than 2 star treatment at 4 star prices at Disney.
BUT the pool is crowded. I spent all of my time at the quiet pool.
Since the OP only has 1 day for the parks, it would help us to know what else is planned.
Like I said, the easiest would be the Contemporary, and with children so young, I doubt the design of the pool matters much - water is all that counts
But the OPs time frame is lower season and I don't think he'd have those issues.