My husband and I used to do the window/aisle booking and often had an empty middle seat between us. We would scoot over to the middle seat if the middle seat had been sold. Now with planes flying at capacity we book an aisle and aisle across from one another. We like to be able to stand and stretch on long flights. I think it is rude to talk around a middle seat pax, but then I have had a very large person try to fit into the middle seat between us. I move to the middle from the window and all is good as long as the armrest stays down.
My wife and I do this often on long haul flights but we often choose to sit together for the simple reason that if we have two seats together there is more combined space than each of us with our own seat.