Ditto what everybody else has said. If all the guy wanted to do was sit with his wife, he would have tried the middle seat option you suggested. The fact that he didn't shows that his real motive was to trade his middle seat for your aisle seat, and he was just using his wife as a prop.