Agreed, however, it doesn't bother me unless I'm trying to sleep.
My point was that there are far more bothersome things in F or BF than a couple of adults switching aisle seats. This leads me to wonder just how bothered they are by the FAs walking constantly between the 2 cabins.