I don't think people travelling together have more entitlement to any seat than a solo traveler. I don't think families should get any sort of priority on the part of the airline, and it is the families responsibility to ensure they get the seats together that they want. In the event of irrops or other unforseeable circumstance, by all means, ask the GA or ask other pax..
But never ever go in with the assumption that a solo traveler is morally obligated to give up a seat for a traveler with a companion simply because you have a companion.
Most of the time, the swap requests are coming from groups without kids...couples, families with members over 12 years old, friends, colleagues. They can sit apart for a few hours without the world falling apart.
I never swap seats unless it involves an upgrade. I don't care about your family. Or your friends. I paid in advance and had the foresight to get what I wanted. I am not giving that up. Complain about common courtesy all you want, common courtesy is taking what you get, shutting up, and not disturbing your fellow pax. Even if you are separated from a child, I am sure someone else with help you out.