I'm actually more annoyed when the people behind you grab your head rest as they're getting into/out-of their seat. But even that doesn't annoy me that much.

Reclining the seat makes no difference in comfort so I don't do it. But I don't care what the people in front of me do.