I never had a problem either until this recent flight.
Window seat has several advantages besides the window and shade: not having anyone crawl over you when they need to use the bathroom, and not having carts and people brushing against you, your elbows and your various bodily members, more "space privacy".
What isn't entirely fair is to have a seatmate who either insists on the shades being opened or closed irrationally and without consideration of neighbors. I just don't think this is entirely right, IMHO
--Richard