I generally assume there has been a genuine mistake - e.g. someone mistaking the seat numbers (they aren't always aligned well) or not understanding A is window, C is aisle etc. I simply say 'Oh hi, is this 4c'? Or something along those lines, as it gives the other person a chance to not look stupid (and, gives them a chance to not make you look stupid!) 'oh is it, oops, I thought it was 5c' or 'ah yup, I'm in 4b - I was just hoping no one was in c' or 'yup, I wanted to use the washroom when the aisle cleared so was just waiting for a chance' etc. before you accuse them of poaching a seat.
If it is like for like, and they are sitting there by accident (5c instead of 4c say) and are settled (stuff in seat back pouch etc) I will likely just switch. If they ask nicely if they can sit there because of a child, spouse etc. and the seat is a genuine like for like (ie not 30 rows back!) then I will probably switch. If they are in any way rude, or I am not willing to switch and they don't like the answer then I calmly ask the FA to assist.