OliverB, No worries. I know how it is with planning.
If the dinner in Nagasaki is a special dinner, I'd reschedule it for another day.
Personally, I would to straight to Miyajima and hit Hiroshima enroute to Nagasaki. Miyajima during the late afternoon and early evening is magical.
I didn't go to Hiroshima castle, so can't comment there. I only spent about 2 hours at the Atomic Dome site, then off to Fukuoka. So, I don't have much to add on Hiroshima.
In my opinion, if you maximize your time in Miyajima and Nagasaki, you'll be very glad you did.
Good luck with your planning. Look forward to hear your trip report.