I could see being a little mad, but only because I would have built up excitement over having the empty seat next to me and learning later that it wasn't.
It's not like the FA was moving people from cabin to cabin, which does annoy me. These were seats that were available just 50' away from the GA to anyone on anyfair.
I'd say *shurg*.