Hard call since it really is between the two seats, and not directly in front of anyone. But I think that both people should get a share of the larger space. Of course, if there were show prints all over his bag when he got it out...
Cheers.
My advice is board first and keep your leg in your half of the middle space to prevent your neighbor from going over the invisible line. If they push against your leg to maneuver it in, do not yield and do not act passively. If the bag is a flexible bag, make sure your footprints are all over it if intruding in your space. It's worked for me in the past.