Well behaved children are probably better flyers than many adults - they will sit and amuse themselves or watch films without disturbing anyone.

The only thing that bugs me is parents who don't control their children and let them cause mayhem with the end result that they disturb passengers around them.

However, given you've taken the time to ask I'd doubt you fall into that category so I would agree - sit where you want to. If you can get the kids window seats and you and your partner take aisles they won't even have to disturb other passengers getting in and out of their seat