Wife + 2 children is one thing, and I have never had a problem at the FL in BOS...but I can see how wife + 4 children pushes the limit. Being one of 5 children myself, I take offense to that, but c'est la vie.
OT, IMO, going from 2 children to 3/4 children, when traveling, is a exponential increase in cost/challenge/complexity, etc.