Some of that is simply the upper classes pretending the lower ones don't exist, because it is embarrassing in a sense for them, and incriminating.
When you add race, it adds that much weight to an already painful issue. The idea that "they" make "us" look bad.
I once heard a recounting of a story of a black celebrity behaving somewhat "offensively" at a private dinner party, like a celebrity kind of thing, and then someone who is black, like visibly dark skinned, gorgeous also said something like
"That's why black people have such a hard time in the business"
but that's not really why, it's because often white people are looking for any excuse to exercise deeply held/sometimes unconscious racial baises.