It is fine if they are sitting in the seat waiting for me, as this can be practical, but if they have already taken off shoes, stowed stuff in the seat-back pockets, or the like, then that is just rude.

As a general rule, though, I try to accommodate, unless they are giving me an awful seat or I would be giving up a great one. I would never trade exit row or bulkhead in coach, for instance, but would switch window for aisle or vice versa. I'm not overly sussed as to whether the seat is forward or behind.

Then again, the last time something like this happened, the guy sitting in my seat randomly happened to be my boss (well, boss's boss), traveling with his kids on a Saturday morning flight. He offered his F seat for my coach seat so he could sit with his wife and kids. I calmly mulled the offer, called him a sucker/idiot/something like that, and moved up front, aglow in my selflessness.

I have asked to switch before, but only when I am offering the other person something objectively better - like my aisle or window to someone in a middle seat, or F (well, intra-European J, FWIW) for Y. Never had anyone turn me down, but I would be cool about it if they did.