Sure they can. And there's nothing you can do about it. What are you going to do - sue? It would be thrown out because you are not a party to the merchant agreement. If Visa wants to let them violate the agreement, they can do so.

I'm glad you talked them into accepting your card, but they certainly didn't have to. The most you could've done is write a strongly worded letter to Visa, which they would have ignored.