The road between Phnom Penh and Siem Reap is fully sealed and in good condition. Taking the bus between the two cities is no big deal, but a waste of time given your overall schedule.

I'd just fly into Siem Reap and pass on Phnom Penh. There are lots of places in Cambodia worth visiting besides Siem Reap/Angkor, but the Angkor sites need three full days.

It's interesting to take a couple of hours on the Tonle Sap lake, too, which works out to 1/2 day with transit time. If there's time, another 1/2 day to Banteay Srei. So, I'd optimize around Siem Reap and eat the added cost. Stay at a mid level/cheapo place such as Two Dragons guesthouse and eat cheap food, easily available, to make up the difference. Ride motos to the temples instead of air con car, avoid taxi mafia at airport, etc. etc.