I'm with you. The couple of times I have been asked if I would mind moving in economy I've ended up in middle seats with one of those times being sat next to a screaming baby. Never again.
Unless there really is a good reason for moving (passenger with limited mobility needs to sit with their companion etc), or if the change does not result in a worse seat, then I'd rather stay put. I was once asked to give up 64A so that a DYKWIA couple (both he and she boarded in sunglasses and were draped in bling) could sit together, no chance.