That's preposterous.

There's no question of refunding you the €20 difference, but to charge you the difference when moving to a more expensive to a less expensive flight is just preposterous.

I think this is a case of hang up and call again - any decent agent should know that you can move in the manner you wish without additional penalty (and without recourse to a refund, which of course you have not asked for!)

(Incidentally - is there no mechanism to change this booking yourself online? It would be quite useful for everyone if they would allow customers to re-book themselves online in ways that would minimize the need for customers to call in!)