I have flown with family members several times where only some of us have been in FC and I always bend over backwards to be considerate of other FC passengers if/when we change seats etc. In my opinion, it's less a matter of rules than an issue of common courtesy.
I say this because it really can very annoying to others in the cabin . . .
A couple weeks ago, my entire family was flying FC to LAX (from EWR). I bought 4 tix because I knew there was no chance for transcon upgrades and just factored into the cost of a nice fmaily vacation. Anyway, there was a couple seated in the last row of FC (directly behind my husband and 12 year old son) who had their 2 children a few rows back in coach. Their kids were very young - under 5 - and I was pretty surprised that they chose to have the kids sit alone! Anyway, as you can imagine, the kids were running back and forth up the aisle into FC, sometimes one parent went back and sat with one child, but for a good portion of the flight, all 4 squeezed into the 2 seats and the kids were essentially climbing onto and banging my husband's seat. I did not want to complain to the FA because as a mom who frequently travels with kids (granted older, well behaved ones), I am somewhat sympathetic. Eventually, one of the FAs kindly suggested to this family that 2 people really should go back to their seats in coach, and I appreciated that.
I realize this was not the case with OP, but I just wanted to point out that the flight attendants have to deal with this sort of thing all the time, and often it really is disturbing to others seated in the cabin but people don't necessarily speak up so it really is up to the FA to "police" the situation.
dc