One thing I do not understand is why airlines could not cater as part of regular meals options that would cover many special meals

For example, a breakfast that involves Gluten Free Cereal, Fruit, and Vegetarian Yogurt would cover Gluten Free, Vegetarian, Low Fat/Low Cal, and a number of others. Lunch/Dinner can be a large salad without croutons/meat/cheese, fruit salad, vegetarian/gluten free dessert. If you need an appetizer with that you can have steamed mixed vegetables with oil/balsamic (squash, peppers, eggplant, etc.). While I would normally grill them steamed probably works better for airplanes. The catering costs would be similar and you do not have the hassle of special meals