I also never use the call button. If it is a true emergency and I need a FA I'll get out of my seat and find one - I'm not going to test their "response time."
If it is not any sort of emergency I'm happy to wait until one wanders by my seat or if I feel really impatient I will go to them and ask for whatever I want.
The only scenario I have seen where I think the button was used appropriately was when small children are being tended to and a passenger needed help but could not leave their seat because of the situation the child was in (getting sick, sleeping on a parents lap etc).
I think the buttons are really a relic of the past.