For me, the key to the discussion is the inconsistency. For most of us, if we know the rules we are willing to follow them. If on some flights you can move to an exit row and some you can't, based on who's working the flight, then it is no longer a rule but a crap shoot. When they are not consistent it puts both the passenger and the FA in a bad position when the 'sometimes' rule is enforced.