I've noticed this maybe once in my 20-30 flights on the UD. Seemed like every time somebody would come out, another one would enter, and combining that with meals and everything, the security fence was up more often than it was down. But agreed that it's not really a big deal to wander downstairs, stretch legs and use the bathroom there.

It would be different if it were primarily, and obviously, social, so it's hard to speak to the OP without being there.