I do that all the time with my wife and two kids. They don't mind. Hopefully you will get a F/A that finds this very charming and will walk you up to an open seat.

That said, I would also talk to the G/A to see if she can upgrade the third seat as well. If there is space, I think you stand a reasonable chance, although it is against the rules.