I have two perfect travel handbags, but the commonality between the two is that they both compress (when empty, of course) to smaller than a deck of cards, so can be easily stashed in my underseat bag.

One is a Sea to Summit grocery tote which is spacious enough to hold darned near anything and is made of a non-staining material.

The other is the world's tiniest hobo bag in a faux suede fabric with a light golden sheen. It is fashionable, but only large enough to hold my phone, a passport, and a hotel key.

I suspect that I use my handbag differently than the OP. I use my underseat bag to hold my travel necessities including, but not limited to, my boarding pass and my on-plane water bottle.

My travel handbag is my purse when I am actually at my destination and have put my luggage into whatever room I am occupying. I use my travel handbag while attending to errands/meetings/sightseeing at that location.