You got taken for a ride, but there's nothing you can do about it now (and very little you could have done then). Best to just let it go.

If it really bothers you, you can always write a letter to the airline complaining that the flight attendant removed you from your chosen seat, apparently due to a baseless accusation on the part of your neighbor, and didn't have the guts to tell you what was going on.

At the time, I would have looked the FA right in the eye and asked clearly "Are you accusing me of staring at this lady's boobs while she is feeding her baby?" It being me, I would have then announced that my boyfriend wouldn't appreciate such behavior on my part.