I was travelling Dubai to Seattle and selected a bulk head seat for advanced leg room. There was a lady travelling with an infant, along with another lady and her child seated in our row. Upon boarding the female with the infant asked me if I could switch my seats with her husband as they wish to sit together. Considering that this is a very long flight and Im very tall (6"4') I politely declined for the extra leg room since the husband was seated further up in a window seat.

Within a few hours of my journey I was told that by the FA that this passenger is feeling uncomfortable because she was nursing her baby. Basically without directly pointing fingers I was accused of staring at her [Moderator edit]. The FA told me that it is in everyones best interests that to relocate my seat and was sent to another aisle seat down the corridor. I politely protested and it fell on deaf ears so eventually complied with her instructions.

However a few hours later I notice that the husband has now conveniently relocated himself to my original seat. I called the FA to show her what's going on but she didn't seem to care as apparently the matter was resolved and still seem to side with the lady hinting that I might have been "looking at her side unknowingly and these things happen." It seemed like she was preoccupied with her duties and didn't bother trying to connect the dots or hear my side of story.

Did I got taken for a ride or was this justified?