If you give the guy on the phone your confirmation number and last name, they can look it up in the system and cancel it. This is also why you are advised not to post details about your itinerary (ticket number, PNR / confirmation number, etc.) on social media sites, etc because anyone can go in and edit your booking then.