Well, here's a different take - if you are only buying a lens for your once-a-year trip to Baja, why not consider just renting the exotic lens you want just for the trip? I don't know where you live, but this option should be available in most major cities.

If nothing else, trying out different lenses and/or lens/teleconverter combinations before you buy might help you with a decision, then you can decide if it is really worth shelling out the money for gear you might only use sporadically.