The museums will be hit-or-miss, and a lot of that will depend on the weather. There tend to be activities on the Mall on most summer weekends, everything from charity runs/walks to protests to festivals, and if it's a very hot day, or there's a pop-up thundershower, the folks on the Mall tend to take cover in the museums. Tourism is generally down for that weekend but we do get a fair number of rallies/marches over Labor Day weekend.
DC public schools start up before Labor Day, as do many of the suburbs, so we do not see the mass exodus for the last hurrah summer weekend over Labor Day weekend-- generally that's the weekend before--- even though it's a 3-day weekend, parents don't want their kids to miss an extra day so soon in the school year, and don't want to suffer through Route 50 beach traffic for such a short trip. So there are a lot more locals in town than one might expect, which goes back to "if the weather's bad, they'll all be looking for something to do indoors." Air and Space Museum often has the biggest crowds (kids love it).
If you are driving to Boston I'd leave on Tuesday (Sept. 4 this year) and not try to hit the road till 10-11am; if you're flying I don't see it as mattering too much although fares do tend to be higher if you fly on the holiday Monday.