There have been surveys of people's satisfaction on a flight. Almost always, when the seat next to them is empty, people say they had an above average flight regardless of whatever else happened on the flight. That feeling of space makes all the difference in the world.

Now if someone is in the E- middle seat next to you then there is little to no difference in my opinion. Then it comes down to a height issue or other factor, like access to the aisle, as the deciding factors.