I always understood it to be that the final "s" is pronounced when referring to the fruit or liqueur but not pronounced when referring to the city.
I learn the fruit name without the 's', but that was in Geneva (GVA), Switzerland. Mind you I was taught by french nuns

(honest).