I would have said the best course of action would be for the kids to move to the seats in the back with their parent. Especially given little kids do not need the leg room.

About the only time I would move is if a FA demanded that I move. After that one gets it documented so to claim compensation for an involuntary downgrade.

Welcome to FT ... stick around there are lots of similar stories. One learns to stand their ground or in this case stay in their seat.