I've in Boston on the 4th a few times, and I don't recall any attractions or the T being jam-packed except for the Hatch Shell area before the pops concert/fireworks. And yes, people start staking out spots for the concert VERY early. My suggestion would be to listen to the concert from wherever you can hear it (one year I literally sat in the middle of Storrow Drive, and I could still hear quite clearly), then right near the start of the fireworks, make your way to the Longfellow Bridge. Absolutely perfect view of the fireworks.
I've done the Freedom Trail on the 4th as well, and none of the sights along the way were too crowded when I did so (three years ago).