It's the proximity of the curtain to the seat. it feels like people are almost in your suite, and they have a full close-up view inside the suite when heading back down the aisle. Invariably they look at the person.

i do it too because they are straight ahead as you pass through the curtain. They need to have the doors on the suite closed to get any real privacy.