I'd refuse if they were asking you to move to a worse seat with no direct access...
This happened once, when i'd been moved without asking, and I noticed this only when looking down at my boarding pass going through security. I went back to the check in desk and they put me back in 64k and I was a happy bunny!