If I were flying alone or with just my husband, then I would take the middle seat. No problem. I've had middle seats, and sometimes the window or aisle guy doesn't show up. Great, I now have what I wanted.

If I were flying with small children, and middle seats were the only ones available, then hell yes, I would look to any other alternative. I would not want to burden other passengers with a child when I have no immediate way to monitor and correct their behavior. I would not want to risk my child being exposed to inappropriate behavior from a random adult seated next to them.