I'd be annoyed should I lose the "Unite the Couple" game, but at the same time, touche to the couple who are willing to sit apart. Win some, lose some... we've all been there on both sides of the table.
Whenever I wind up in this situation, I keep in mind that the couple chose not to be seated in the middle, more power to them... I gambled with the assumption they'd want to sit together and voluntarily picked that specific row... and lost.
Probably a bit grumpy over it, and certainly more sensitive to the back and forth passing of items. Should it annoy me enough (working on my laptop while they pass things back and forth) I'd ask if they wanted to sit together.
If they choose not to, there's little recourse except maybe ask for the FA but I don't think I have ever gotten to that point. I figure once I am in that seat, the immediate area in front of me is my personal space and I shouldn't have to be subjected to constant disruption because they want to go back and forth. I won't mind much if I wasn't doing anything, but if I were reading or working, it would be a concern.
For what it's worth, the couple "shouldn't" have to be forced to sit together... they have as much right to sit one seat apart, or an entire aircraft length, should they choose, and I certainly am not in the position to suggest the scoot together so I can get the window or aisle.