I've seen a passenger in the exit row with her kids in the row in front of her (sitting with grandparents). At cruise, the kids came back to the exit row, and the F/A promptly kicked them out of the exit row, saying that they weren't allowed to be there at any time of the flight.