OP - Grab pickup location in SGN is not at the curb side in front of the arrival terminal, but I believe (i have not used Grab from the airport to town) it is located in the parking structure across several lane of traffic from the arrival terminal building.
I speak the language and I still prefer to catch a taxi from the airport to, as you say, get on my way quickly to my hotel.
Even with your late night arrival, taxi is not a problem and the ride to the Hilton should be an easy 15 min ride at most.
In Feb 2024, the ride at 11am cost me around 230,000 VND to the Grand Hotel, a block from the Hilton..
Other than from the airport to my hotel, I use Grab or Xanh for all my local transportation.