I think it is quite odd to move into a middle seat in such a fashion. As I understand it, you have no official claim on your wife's seat. When she didn't board, she lost the right to the seat. The airline could have assigned the seat to someone else and there is nothing you could do about it.
Having said that, I would have considered telling the guy that you paid for an extra seat, and that he can't take the middle seat. Even though that isn't true.