It's a drive - done in the daytime, should not be problematic IMO - I've done it several times. But just remember, it's Mexico, not the US, so animals on the road, etc. are not unusual. Budget plenty of time - 6 to 8 hours is probably normal, but you might find yourself stopping for some great sights and places like San Martin Tilcajete (for their fantastical alebrijes) and Santa María Atzompa (green glazed pottery).

Myself, unless I were stopping elsewhere to sightsee (Puebla, the villages in Oaxaca en route, etc.) I'd rather fly (or take a decent bus service, see above) to OAX and rent a car to visit many of the interesting villages like San Bartolo Coyotepéc, Teotitlán del Valle, etc. as well as the archaeological zones like Mitla, Monte Albán, etc.

Oaxaca is a very diverse and culturally rich state.