You have a right to the seat you reserved/paid. I have moved in F before to accomodate a couple. Feels good to help someone out. But, I have also declined when I felt the move wasn't a fair trade. Example, was once asked by a couple to move to 1E on a domestic 757 from my row 3 aisle. Sorry, I politely declined explaining that 1E had restricted legroom because of the bulkhead. I didn't feel a bit of guilt in doing so in that instance.
In Y now I always book a MCE or Pseat. So, any expectation that one would move from those specifically selected extra fee seats is easily defended.