Yeah, there is still racism in some areas but as you mention it isn't much on display in areas that are used to tourists. I think you will find that kind of thing anywhere in the world. It sometimes has to do with a fear of having to communicate with someone who doesn't speak the language but other times is just a dislike of someone different. Speaking the language might diffuse the situation but also might just embarrass the person and make it worse. I usually just do what you did, politely leave and never come back.