Any food of which people say, "Oh, you have to try it with (side dishes, condiments, a certain alcoholic beverage)" doesn't actually taste good.

Caviar is definitely in that category, as is lutefisk ("Oh, you should try it topped with melted butter while drinking aquavit.")