I've gone from the second to last gate in E to B then taken the bus to A. Now I've got to the C bus gate before where half of the passengers for those flights are standing in line, and they hold the plane to Waco or College Station. In that instance they've taken the bus directly to the waiting aircraft, bypassing the gate. In that instance I've spent more time waiting for the bus than from walking from E-C.
I think you're safe.