I believe the official rule is that is not allowed at all. I have even heard of UA bringing police on board because of it, I remember a flight SIN-SFO where that happened and a husband and wife were switching between seats and carrying their kids with them, etc.
If I was in your position I wouldn't have put up with it for a few hours, that's for sure.