It is a tough situation, with no real way to win. These days, it is not hard to get arrested after landing if you ignore the FA's demands. Especially as a man and involving any accusations of sexual inappropriateness. You just can't win after being accused of anything like that, your best hope is to not lose big.

I guess your best chance was to be very polite, very quiet and friendly and simply ask for the purser and explain your side. But at some point, right or wrong, if the FA's tell you to do something you have to do it, or risk an arrest after landing.