Sep 3-4 should be fine imo. We've done it a bit later (would have been around the 6th) and at that point only about a third of the huts were still open so while your choice is more limited there are still people on the mountain. That night there was probably only about 30 people climbing the mountain and our small group made it to to summit first and had about 20 minutes there by ourselves which was a bit of a buzz.

All that said, if you are travelling light and don't have cold and wet weather clothes do not climb Fuji. It will definitely be cold and has a good chance of being wet and if not prepared you can find yourself in trouble very quickly.