This is an impossible question to answer, since what is 'racist' and what isn't is one of the things the question must address in the first place.
On a separate but not entirely unrelated theme, I watched an interview with a displaced family (Afro-American) from New Orleans who had been relocated to Utah. The mother in particular noted that 'everyone here is white' and admitted her discomfort, saying that this could never be home.
If she had been white and from Utah and had made that comment about NO and being uncomfortable about living where 'everyone is black', would she not have been branded as a racist? Would she have received the sympathy that she was clearly getting from the interviewer?
More likely human attitudes are complex and variations cannot be explained on the basis of either genetic (skin colour) or subjective (i.e. 'racist') reasons. That we fear the unknown isn't entirely a stupid or undesirable response. There are advantages to proceeding cautiously in unfamiliar situations, and to deride that caution as 'racist' in either black or white or whatever is to demonstrate an absolutist 'all-or-nothing' type of illogical thinking.
It's only been a few days for the Afro-American family in question. Meanwhile, I see their apprehension not as 'racist' (an impossibly general and subjective term to me) but simply caution taken to the max. Their personalities will likely decree how they proceed in the future, more than their skin colour. Mature people will come to see a new experience as just that, a new experience, not necessarily a life decision, and will relax somewhat. Thus, in time, while this family may still wish to return to NO some day, they may find that while some of their fears were well or even only somewhat well-founded, those of white skin colour are a very diverse bunch and not necessarily impossible to live with.
Or even to visit. And herewith the answer to your question.
Am I cautious about visiting cultures different from my own? Yes, but the degree of caution varies with the degree of difference that I perceive that culture to be. Visiting Europe I perceive as less different an experience from visiting Japan, and visiting either as less different and stressful than visiting a place where war has recently or currently beein in the picture (Middle East, much of Africa, etc.) In the highly emotional climate that typifies cultures at war, I could be a target for no reason at all, because emotion doesn't accomodate reason very well.