I always understood it to be that the final "s" is pronounced when referring to the fruit or liqueur but not pronounced when referring to the city.
I'd say the same... but local variations are always possible, I guess.
Good thing we have fewer local accents in France than types of cheese
Cheers,
J-