Himeji is a very pleasant town and the tree-lined walk from the station to the castle is relaxing, but it's not exactly brimming over with tourist sites.

Unless you just want some time to chill in a low-pressure town and see a slice of ordinary life, use Himeji as a stopover. Be sure to leave Hiroshima as early as possible, because touring the castle means climbing a lot of ladder-like staircases to the top lookout (which affords a fantastic view of the surrounding area), and I would hate to do that on a typical July afternoon.