It happens sometimes - reasons include power-tripping FAs, lack of onboard FA supervisors (on U.S. airlines), FAs not properly trained in conflict resolution and other public-interaction skills, the terrorism bogeyman, overuse of "safety" as a vague but easy excuse. Captains will usually side with the flight crew because it's in their personal best interest to do so. Though the Captain's authority to refuse to transport is supposed to be bound by certain legal parameters, but it's easy enough to cite safety based on the thinnest of FA complaints - and there seems to be little to no review of such decisions by airline management.

See for example

https://papersplease.org/wp/2012/08/...t-let-you-fly/

https://papersplease.org/wp/2012/08/...o-let-you-fly/