I witnessed this once on another airline.

I was in the exit row and one of the flight attendants realized that a gentleman in the exit row was traveling with his wife and small child who were seated several rows back.

I heard the flight attendant explain that in the event of an emergency, his priority would most likely be on assisting his child and not opening the door or assisting anyone else. Therefore, she made him move.

As someone else said... it is not a matter of you must sit with your small child. It is a matter of you can't sit in the exit row if you have a small child onboard.