I did something like this two years ago on CO 62, EWR to MAD. I was upfront with an upgrade and wife and three children in the back. No, I'm not a selfish pig, I got an Op-Up and my wife said "take it." Because it wasn't right to leave her in cattle class with three little ones, after takeoff I took my smaller daughter, age 3, with me in 7D on the 764. That's the seat by itself in the middle, closest to coach.

Children being what they are, the others wanted a turn with daddy and after the meal, each got about an hour in the big seat with me. We did the switches very quietly and there was no running in the aisle, shouting or even discussion.

The only comment from the FA was as we were approaching MAD, that my son would have to go back to his own seat for landing.

Based on my experience I think if grandma and grandpa sit in row 6 or 7 of BF, and your little one is not all over the place, you should have no problem.