I don't know how old your kids are and whether they count as "small children", but here's a google result from FAR Part 121 Sec. 121.585
(b) No certificate holder may seat a person in a seat affected by this section if the certificate holder determines that it is likely that the person would be unable to perform one or more of the applicable functions listed in paragraph (d) of this section because--
[...]
(7) The person has:
(i) A condition or responsibilities, such as caring for small children, that might prevent the person from performing one or more of the applicable functions listed in paragraph (d) of this section; or
Maybe you've been lucky before, but it makes perfect sense to not allow an adult whose first priority is going to be their kids and possibly interfere with others trying to evacuate to not to sit in an exit row.