Does it happen? Yes. Do you have to let it? No.
As a passenger you are entitled to direct the route taken. Every time I've received push-back I just repeat my instructions and make it clear to them that I am choosing the route.
I have, on occasion, also let the driver know that I actually live here rather that just being a tourist. That also helps on occasion.