Generally I would agree that the agent was wrong but the fact he called up to ask and asked in a neutral way meant I had no ground to stand on. I remember the name from the agent's badge but I think it is not the done thing to refer to it here.
I'll write a complaint as you guys suggest because either way it is not great service. I'll let you know if I get any Avios!