I so understand the OP's story. The "we will come and get you for boarding" aspect of TGs ground game is not really well communicated.
And I get REALLY antsy about it as the moments tick by. I guess it is all that United Airlines angst about being mistreated that gets to me. Lately, I just let them know that I want to stretch my legs and wander to the gate on my own time.
I do enjoy the walk before the long-haul flights, plus getting settled in with a glass of Dom is relaxing to me.