Have you started the itinerary already? If so, then it seems the agent is following the t&cs outlined here

Specifically:

Where changes are made to the date and time of the Upgrade within 24 hours of the scheduled departure of the first flight in the Reward travel itinerary or after the scheduled departure of the first flight in the Reward travel itinerary and the redemption class for the Upgrade is not available for the amended flight(s) Avios Points will not be re-credited to the Member’s account.