Leave it up! If the person next to you wants it down, then work out a compromise, including the fact that aisle seats are better than window seats except for the window.
If the FA wants you to close it because someone else in the cabin wants it closed, other than the person next to you, I say too bad. How can light from one window possibly irritate someone who is sitting in a different row? (excluding direct light, that's understandable). For heaven's sake, have a couple drinks if you're having so much trouble getting to sleep.