I did this once. I first asked for permission from the driver as I simply wanted to go near where the property is. It might be possible that he recognized me from past stays, but I wasn't sure. He had no issues with it. I gave him a bigger tip than I normally would as a guest of the property.
I guess I'm going to hell based on other responses here. ¯\_(ツ)_/¯ How dare I!