Drive time between Monterey and Hearst Castle depends on which route you take. If you go on 1 it will take much longer than if you're on 101. 4 or 5 hours on 1 seems quite a bit high though. Although it will depend on how much traffic there is (can be much slower during summer with all the RVs), I would think it would be closer to 3 hours, if that. Going on 101 will drop it closer to 2 hours.