I don't think it's a stretch at all; I've seen this happen many times. AA's FAs routinely catch families when one adult tries to sit in the exit row while flying with their spouse/partner and their children (under 15 years old).

Very unprofessional, IMO, for the FA to fail to explain to the OP why he was ordered to move. While "unreliable" may very well be FA jargon for such a disqualified person, there's a bunch of ways the FA could have handled it more professionally. Announcing that he was "unreliable" to the entire cabin borders on defamation, IMO.