Yes, its okay. I would encourage you to trade them for the whole flight. On my way home from Orlando last week, there was a Mom and Dad in FC and a gaggle of kids (theirs?) in coach.
I was in FC and it was rather annoying to have them sending one kid back and having another sent up every 15 minutes.
I didn't say anything because it was only a 2 hr flight. (MCO-EWR.) Had it been a transcon, I would have encouraged the FA to intervene.
After all that negativity, let me say something nice. Giving up your FC seat to a friend, family member or even stranger (an elderly individual, or a Service Member in uniform) is a very kind act.
Remember that (at least with OnePass) you can upgrade a traveling companion if you are Gold or higher.
Then both of you win!