I've had similar situations with elbows in my seat space, but never experienced a reaction when I kept my elbows inside the armrests. I find it very frustrating when I'm in a middle seat and the person in the aisle/window takes the armrest and has their elbows over the edge of the armrest. The passenger in the middle seat has no place to move.
I guess a lot of people forget about the Golden Rule.