Why would you want to force the guy to switch? If his travelling partner was seated beside him, than you would likely get the "stink eye" for the rest of the flight if you forced the issue.

I'm with the others who commented that the problem appeared to be the unruly family/children seated behind you.