That has happened to me a few times also where I have lost the empty middle seat next to me when a co-worker moves from E- to E+ to sit next to his/her co-worker in my row. One time was a rather large person.

They have been short flights either LAX-LAS-LAX or BUR-DEN or LAX-DEN. However, I still would probably not say anything no matter how long the flight.