Ultimately the position of the window blind is the prerogative of the cabin crew. Until they get involved though it belongs to the person in the window seat.
The passenger who ignores requests from the cabin crew to lower the blind should also understand however that when sitting in the aisle seat it is my prerogative to stand up when you need to use the lavatory.
If you flood the cabin with sunlight after the crew has asked you nicely to lower the blind I'd suggest you go easy on the coffee or have sufficient dexterity to climb over me and the tray table when you need to pee. Otherwise keep your legs crossed and enjoy the view.