Saddle Road is more annoying than scary to me. It's bumpy and too narrow, and it takes a good long while to get to the road to the observatory. (BTW, I think the observatory road is about midway between Hilo and the Kona coast).
I agree with the others about altitude effects. I am very athletic, but I got winded just walking around the telescopes. Perhaps you can get used to it, but I wasn't.
As for a night trip. Hmmm... Is there some way for you to get a 4WD? You can probably do the gravel part of the road in a 2WD, but a 4WD would give you some peace of mind, especially at night.