Just let out passive aggressive harrumphing vibes and ensure you unequivocally own the shared armrest.

I think generally empty middle seats in exit rows get grabbed for staff travel quite often. I've started to avoid them, as blocks tend to stick better in other rows.