Sure it has legroom, but armrest space and shoulderoom have value too.
At 6'3", 240 lbs, I can squirm my legs around under an empty middle seat, even in E-, and my butt fits fine even into a 17" seat, but with a 48" chest I need more upperbody space than I get with 3 across.
Right, but some people are 6"6' and 180lbs, in which case they may feel differently. I certainly don't think it's a no brainer to take the E- aisle with an empty middle, but that's just me. Sounds like he was hoping you would move.