It's completely safe, as long as you don't fall asleep on the wheel (the desert is quite dark on the drive). I actually prefer to drive late night as during the day, the scenery can get annoying - the desert is only so interesting.