I'm usually willing to move if the people asking are using common sense. I think
chemist661 said it best...common sense is only asking to trade if the swappee is getting a like or better seat.
In a J configuration, that means I'd attempt to have 1 window and 1 aisle in hand before I started thinking about trading. If it was a 2-3-2 and I had two middles, I'd just deal with us sitting apart. (On a 2-4-2, maybe you can get two middles together.) Two grown adults
can fly on an airplane separated by a couple of rows. Both will arrive at the destination at (approximately) the same time.
Part of the deal with gate upgrades is that you sometimes get middles in J. That's okay - think about the alternative, sitting in coach.

I'd rather my spouse and I enjoy separate middles in J than seats together in Y, every time.
Always watch the J-to-F upgrade list at the gate. If there appears to be some movement there, let the GA know you're interested in getting 2 together in J. That happened for us on a TPAC about a month ago. We held two J seats apart, then about 8 people were upgraded to F, and because we'd mentioned it to the GA she moved us to two upper deck seats together before she started working the Y-to-J upgrade list.