I understand its more than annoying when somebody takes your seat, I also play middle seat roulette (aka BA buisness class) from time to time. Usually the worst case is someone turns up and we usually ask them if they would prefer the isle or window and do a no fuss swap, but it sounds like you and your wife do not want to sit together (eg you its not the empty seat your after - its a seat separating you and your wife?). Perhaps if you told the cabin crew this they could have accommodated you in another row rather than sit next to your wife and everyone wins?