I have never heard of a normal ATM charging a commission although some charge a fixed fee.

ATMs attached to a retail bank do not, to my knowledge, charge fees (never seen one that does). Other ATMs sometimes do but they tell you before the transaction. Typically, there will be a screen that says something like:"You will be charged £3.50* for this transaction. Do you want to proceed?" and ask you to press Yes or No. I think that this is a legal requirement for them to give you notice of the fee although I am not 100% sure.

The only machine I can think of that charge commission are change machines sometimes encountered at airports. Could it be one of these that you used?

*: amount is made up. As I avoid these machines, I have not memorised what the typical fee tends to be.