I find it strange that people don't feel comfortable refusing. Someone asks you a question, you have the option to answer either yes or no. It would be silly to answer and then grumble about it to yourself after.

No harm in asking, no harm in declining. Any adult should be able to do either with no fuss and no hard feelings.