Actually I saw this happen to someone else.

There was already one gentlemen sitting on one side of the exit row (1 seated, 2 empties).

The flight was relatively empty and another gentlemen attempted to sit by himself in on the other side exit row (3 empties trying to convert to 1 seated, 2 empties).

They told the gentleman he would have to move. He moved back to his seat one row ahead.

As the airplane started to move, the gentleman moved back to the exit row (I know he should not have done this, but ...). This time the flight attendant came very rudely and told him he would have to move or the airplane would go back to the gate and he would be removed.

What's the big deal? Why not let him sit there?

Any rule I don't know about? why?