I don't believe the app has my home address saved (though not hard for them to figure out). When calling Uber/Lyft I may enter my home address as a destination, I usually use next building just in case). Of course, in my case these are multi-unit buildings right next to each other, so that the location is physically the same.

TBH I am more worried about random "delivery drivers" Amazon is using for their delivery service lately.