Well, the airlines do tend to side with those who bring the animals on board ("allergic? too bad, you can take the next flight if you want"), so it's a little inconsistent that they wouldn't also side with the person that brings the nuts on board.

I do wonder what would have happened if the OP went to an FA and said "I have my own dietary restrictions, I can't eat most of the food you're serving, and thus I responsibly brought my own food on board -- but it contains nuts. Am I supposed to starve because someone somewhere on this flight claims to have a nut allergy?" I wonder if the FA would have "hinted" that it was just a request not to eat the nuts and they can't force you.