Yes, it has happened to me several times that my seat was taken by others.
One young man, who had taken my seat, generously pointed me to a middle seat several rows over. I refused it and insisting on taking my assigned seat. He would only budge when the flight attendant approached.
In another instance I also got the reply (seems standard), 'there are plenty of empty seats you can take.'
So far I have not caved in to that.