I had my first taste of caviar in 1987 in Moscow and then in Leningrad (now St Petersburg). Many of the people who I was traveling with felt the same way you do. I gorged on caviar for the next week and always loved it.
It's a combo of the quality of the caviar and your tastebuds.