Safety first of course, and it's absolutely right that a pilot who feels they are unable to operate a flight safely should decline to operate that flight. But I don't blame passengers for being annoyed. I think most passengers would find that to be an odd reason for a flight cancellation.