First, if you do drive at rush hour, stay off 95 if you are in a car. 95 is often backed up from exit 2 to 27 (Greenwich to Bridgeport), take the Merritt which you can usually cruise at 70 most of the way to Fairfield, with slowdowns just a couple exits there.
You can cut over in Milford to 95.
that being said, and I live right off the Merritt, rush hour is not the time to be taking any of those roads. If you are talking Friday rush hour, that's even worse. Holiday Friday, well forget it.
Make a reservation, take the ferry to New London and save yourself potentially hours and hours of stop and go.