I'm guessing you're using android? On iOS the app has never asked me for access to contacts and it works fine without location access.
I haven't used android in a while but my memory is that it's similar to iOS in that you can deny an app's access request and the app will still run (with, perhaps, reduced functionality). So just ... don't let it access your contacts?