I did the drive for the first time this past October. I left Fort Lauderdale on a Saturday afternoon and drive down via the Turnpike to Hwy 1 and didn't hit any serious traffic; I think we were in Marathon in a little over two and a half hours. Overall, I was pretty underwhelmed by the drive from Key Largo to Marathon but we did enjoy our time there (stayed at the relatively new
Hyatt Place) and ate at several nice places like Burdine's and the Lighthouse Grill. The drive from Marathon to Key West (we also stayed at the
Hyatt Beach House) was much more scenic and enjoyable. It took about an hour without much traffic, IIRC. One of the days in Key West was when a cruise ship was in port and it was a bit of a mess, so I avoided Old Town that day. We had very nice meals at Little Pearl, El Siboney (Stock Island), and Hogfish Bar and Grill (Stock Island).
I drove back on a Tuesday via the Turnpike to the Reagan to the Don Shula, then the Palmetto and finally the Dolphin into South Beach and I think it took about three hours (we stopped and took many photos along the way). I ended up
buying my own SunPass to pay for the tolls rather that the marked-up PlatePass fees from Hertz. I don't know if I'd ever want to do the entire drive again.