I can deal with small businesses not taking Amex - although I suspect my CSR costs them almost as much.
What really tees me off are businesses that are cash only, and have an ATM for my 'convenience'. Seriously, they won't pay a 20-25 cent commission on my $10 order, but will gleefully accept $2-$3 for me to use their ATM.
Maybe this has nothing to do with the credit card commissions and more to do with keeping revenue off the books and tax returns...