So the middle seat is empty. What are the do's and dont's of using the middle seat as storage and using the middle seat tray. At what point do you stop the invasion, if at all?

I'm often lucky enough to have an empty middle seat on one leg of my weekly return journeys. Sometimes I just place my phone down onto it, nearest to me, as I listen to music. However, there have been times when the window-seated passenger has put the contents of his/her office desk onto it and then proceeded to occupy all of the tray too. But, usually, there's a tiny bit of sharing going on which is perfectally agreeable to both window and aisle passengers.