It's easy for me to to sit here and say if it's not me in the 'hotseat' at the time, but the bagmover deserved to be told quite sternly, if not a little harshly after the intial inquiry, to take his kids and vacate the OP's seat.
However I appreciate it can be a difficult position to be in. I certainly don't seek confrontation but there are occasions where it's required.