Yes you have, you are just associating the word with food that is usually salty rather than sweet. 'Savoury' can also mean full of flavour or having a rather nice smell e.g. like apple pie