This brings to mind an incident where the window pax said to me, as a means to get to my coveted aisle seat, "It's probably better if we switch, because I have to get up a lot during flights--you'll be more comfortable". My answer: "Well, I have to get up a lot during flights, too, which is why I made the extra effort to check-in early and get an aisle seat. I won't mind if you have to get up"
The window pax didn't even get up once during the flight.
If a COS used a "We'll both be more comfortable..." line on me, I'd would just counter with the truth--"Well, no, we wouldn't. I need the armrest down, please".
It's a pity, but I bet many infrequent fliers would indeed be too afraid/surprised etc. to say anything.