An arbitrary 4-hour rule is dumb. They need to be flexible.
I was on a weather diversion that kept us on the ground over 4 hours waiting for fog at JFK to clear. If we cancelled at 4 hours we would have missed a chance to go home. Even if we just went to the gate, that would have only made us wait longer. It was better to stay on the plane and get home.
Furthermore, if pilots take off too soon hoping to beat the arbitrary 4-hour time limit, they run the risk of continued bad weather forcing a second diversion, and that would really ruin your day.