If it didn't interfere with your crew's collective ability to do their job and it was approved by your manager, I don't see the issue. Did you get any comments about WHY they thought this was an issue? Did you make a point of mentioning that your son was flying with you? If so, this is the one thing I might not have done in your shoes. This isn't anyone's business but yours.
Just my 2 cents...