Let me say that I believe that people have a right to the seats which they are assigned. I certainly am not going to switch my window or aisle for a middle, not on any day.

But to answer the OP's question, on a personal level I find interaction with strangers somewhat stressful and unpleasant, and I find extended physical touching by strangers downright loathsome. I find interaction with people I like to be highly pleasurable. So maybe the reason people want to sit next to people they like is so they can have a highly pleasurable experience rather than an intensely unpleasant one.

I like the fact that more airlines are selling premium seat locations. People who want to sit together can pay a bit more and do so. And it gives a perfect excuse for not exchanging seats with the mother of a child: "Ma'am, I paid extra for this seat because of its premium location, and somebody would have to pay me an awful lot to move out of it and into a middle seat."