Why do people keep trotting out the "if it bothered you then it's your own fault for not confronting them" line on things like this? You are aware that many people on this planet are rather non-confrontational by nature?
While I agree that the FA's aren't mind readers so you can't expect them to know it's annoying you, it also doesn't mean that it's all your own fault.