It's infrequent that I've seen someone moved out of an exit row, but I've almost always seen FAs do it when necessary. Examples include younger children, people who don't understand English (on a domestic USA flight), and those with visible handicaps. I've never witnessed someone moved for ignoring the safety briefing... the FA's typical warning of, "If you don't pay attention I'm going to have to reseat you," has always worked.