I agree. The key element is whether one reseats themselves (i.e. poaching) or one is reseated by or with the assistance of the FA's. In the former case, IMO, it's reasonable to take up the issue directly with whoever is in your seat; in the latter, the FA's should help sort things out and you should be able to avoid a personal confrontation with a pax who's probably not responsible for the problem in the first place.

IMO, it's a good idea to know which situation you're dealing with before lighting into someone too hard. Although if they are poaching, I say no holds barred!