I always understood it to be that the final "s" is pronounced when referring to the fruit or liqueur but not pronounced when referring to the city.

Unfortunately I don't have a definitive reference at the moment, so I could be wrong...