On takeoff and landing I'm glued to the window, especially when I'm flying into/out of somewhere I know well. However, often during the middle portion of the flight, the ground is obscured by cloud cover. Or, it's nighttime and you can't see much. Also, I might want to read/use my laptop/watch a movie and the glare would detract from it.

I do like to occasionally lift the shade to see if I can recognize anything, though from 40,000 feet, it's often hard to do so