The single answer to why we struggle to sleep that there isn't a single answer.
This article gives ten sleep solutions one for each of the five senses and five key organs.
Like all the best ideas this holistic approach is both brilliant and (in retrospect) obvious!

https://www.sleepmastereurope.com/blog

I've never really solved this mystery, even in Biz. The only flight that I can recall getting some decent sleep on was LAX-SYD, in economy plus (or whatever UA called it at the time). I had an aisle seat, the middle seat was vacant. I put small headphones playing soft jazz OVER wax earplugs. Slept about 5 hours.

Sound is my big problem. I've gotten some noise-canceling headphones that help a lot, but they're so clunky that I can't lie on my side. Melatonin seems to help as well (especially with adjusting to the new time zone).

Let me know if you figure out the magic bullet.

I thought I was well prepared on my last midnight flight from LAS to YUL. I had the window seat in business class (with the shade closed), noise cancelling headphones and a light blanket. Unfortunately, every time I opened my eyes, I saw the glowing IFE screen from the row ahead of me through the space between the seats, so I'm adding an eye mask to the list for next time. I didn't get a good sleep but I managed to get enough rest to help me get through the following day without crashing.

I have never managed a decent sleep on a plane. The best I've ever done was when I was flying Europe to the US and our connecting US flight was delayed. I was so exhausted at that point, my head kept falling forward...

Just did an overnight flight on flat bed. I couldn't sleep a wink. If there was a way I would love to know too.

"A little" and don't take strangers' advice on drinking!
 

When I flew from Shanghai to ATL a few years back by business class, my fellow traveler admonished me for ordering a glass of wine. He suggested getting a bottle of wine. Stupidity took over and I took a tablet (without knowing what it was, despite being a medical scientist) that was supposed to make me sleep better. A bottle of wine and unknown tablet later, I had the most miserable flight. Couldn't sleep, ached all over, and felt nauseated for the entire duration of the flight.

Me, too. I sleep on the plane from PHX-FRA, but I couldn't see sunrise during my flight. They asked passengers to shade down. I wish I can see beautiful sunrise. It's gorgeous.

As for my return flight home, I slept all 3 empty seat. I flew out FRA-PHX on Condor. I was so extremely tired. It's a very long flight home.

Apparently, our brains go into a kind of "half-sleep" mode when in an unfamiliar place. This is a defense mechanism that probably goes back to our earliest mammalian ancestors. This is largely why people have trouble falling asleep in hotels, particularly on the first night. As they get used to the place it gets easier to sleep.

On a plane, we feel quite vulnerable, and we are surrounded by strangers in a perilous situation. No wonder most people don't sleep that well in the absence of drugs and alcohol on a plane.

I had something like that. It was in a J cabin but some guy three or four rows up had a lap top open and for some reason every now and then the screen would lite up and the bright flash of light woke me up every time. I don't explain why I was not wearing eye shades. I also never figured out of the guy was waking up and going back on his computer or something was just jarring it.

Another horrible flight experience had to do with those "remotes" for the entertainment, etc., that in certain seats were stored down along the armrest and people would constantly be bumping, rolling into, the flight attendant call button. I mean about every ten minutes on a TATL. The crew didn't seem to be able to stop it. It was on some Delta flight or another years ago.

A glass of red wine. A glass of whisky. Close my eyes. Good night.

I can fall asleep before the plane starts to fly))))

I was in an AA J class lie-flat on a transpac recently. You would THINK that it should be easy to sleep with a full lie flat, pillow, sheet, blanket (I never wear the PJ's they offer), ear plugs and eye mask...but the fact is the flight was bumpy so it was impossible to drift off. I was feeling slightly queasy from it..and the air was too hot in the cabin..had been all flight. And sitting on the window side, I had a clear view of the fluorescent bulb that runs from front to back. I think I probably did manage to squeeze out maybe an hour of sleep eventually. But it was a 16 hour flight and I needed way more.

I can usually get a quick doze on short haul flights nowadays. I used to be able to sleep like the dead in economy. The tray table size, seat pitch, and my body size were all in proportion such that l could actually cross my arms on the tray table and lay my head down on them and sleep.

Not anymore.

On long haul flights now, there's a lovely combo of a mild sedative, a bit of wine (not too much - alcohol really messes with my ability to sleep if I consume too much), and don't over eat. And angled or lie flat seats, can't be too warm, wireless noise-cancelling headphones with something playing, eye mask, and change into something comfy (at the very least, the bra comes off).

I'm a fairly fussy sleeper but require an odd balance of disturbance and peace. And l have a circadian rhythm disorder so l have a truly bizarre relationship with time zone changes (mostly one that consists of IDGAF and l have no internal sense of what time it is when I'm traveling nor does it bother me in the slightest that I'm often eating meals in the "wrong" order)

Based on your profile I see you lived or have lived in places with no-daylight winters and no-dark summers. I always wondered if that helps with Jetlag, or completely makes it worse.