Depending on where you're going, you might find you may instinctively react to something you would deem racist in the USA but can then use rationale to put your experience into context.
For example, had a young girl I was talking to in Asia say "what's up my nikka" to me one day. Had to tell her that despite what she sees in American movies, Black Americans don't all appreciate being addressed like that. She apologized. When she said the phrase, though, I found in the instant, my body reacted but I felt no anger because I understood I was in a homogenous country, talking to a young girl who had never met a Black person in her whole life, American or otherwise. I didn't mind teaching her. She was truly ignorant.
The same thing happened to me when I lived in North Africa, this time, on the street while I walked past some young boys who didn't even speak English but who seemed to be able to tell I was American, perhaps. That was the first and only time Ive ever been called nikka in public - so ironic it wasnt even on American soil. I was angry initially, even tried to hold on to my anger but soon let it go when I realized it was similar to the first situation. They didn't know any better and because of the language barrier, I couldn't really explain it to them. They were so young, though, it was obvious to me they were just repeating something they had probably heard somewhere and weren't using it the way someone might here towards a black person.
Found when I'm traveling somewhere where the people either are dark like me or at least have color, I'm more willing to relax and even try to talk to someone who is curious, even when their behavior initially strikes me as odd.
Never been or had the desire to go anywhere where the population is majority white though...at least not outside of a layover. I imagine I would be as on edge as you described in your OP.
Where are you traveling to, btw?