Nothing you can do about it if you've tried to book aisle and can't -
But the honest truth is - while I wouldn't say a word, and would comply each time asked, it would irk me, for sure, to get up ten times during a flight to let someone out.