i'm sort of a big guy (6', 210) but i always give up one armrest.
I do hate when people invade my space though....for example, on a flight from atlanta to iad, this guy next to me was reading the paper and kept putting half the paper in "my space".....it drove me nuts