Oh no, there are tonnes worse than this

But I agree with the majority view here. Because the people sat in the middle and the window have no opportunity to move until the person in the aisle leaves, you have to give them an opportunity to get out too, rather than just (rather rudely IMHO) assuming that because you have managed to fight into the aisle, you get to keep them pinned in their seat.