Can you replicate this with another device? It sounds as if the Uber app on your phone may not have accurate location data.
Uber uses the phone's Location Services, same source as Google Maps. If you turn off location, mobile Uber wouldn't work at all. I could still get Uber price estimates without location from their website on a desktop.