I've had similar "incident" once on HEL-CKG. At check-in as AY Gold I had asked the seat next to me blocked (row 41 center bulkhead, I had an aisle 41D) for personal comfort. After boarding was complete a chinese couple came around and asked if they could sit there in the middle bulkhead. I politely tried to avoid DYKWIA and tell them that I understood this seat is blocked at check-in. As a response to this I got only "Please?" and so made they their way to 41EG.
I rouse this issue with one of the FAs and I asked is there any other seat I could have seat next to me free. She went through her PIL and noted that seat blockage is mentioned there so it should be honored. Also Y purser came around and told that they will deal with this after takeoff. And indeed then the Chinese FA at L3 bumbed the other one away leaving 41E free.
On a short flight I would have maybe stayed silent but on the other hand I think people should stick to their assigned seats as usually there is a reason why seats are assigned and why some seats are left free/blocked.