Slightly OT but I feel like I'm prey once I'm onboard aha

As a solo traveller I've been asked quite a few times if it'll be okay for me to move seats as a couple have been split up for example. Normally I'd oblige but If it's an exit row seat for example which I've paid for, or a window seat and I'm moving to a centre middle seat, I politely decline and get a bad look! :'(