Agree that it shouldn't be a problem, but if you really think it will, take some stuff out of the other bags and put them in the spare, now it's no longer empty.

What we do is pack a fold-flat duffel inside one of our other bags, it's just easier to schlep around that way.