I suppose it depends on the conditioning.
I'm sure a group of white guys walking up the street in South Boston would make some people just as uncomfortable as a group of black guys in Brownsville.
There's no denying that history and media has conditioned many Americans to view blacks with fear, though.
I like to think that people won't do anything to you unless you provoke them (generally, anyway).
I am kind of reminded of that old Chris Rock bit, though. When he was talking about the Columbine tragedy.
"I saw these two white boys hop on the elevator...I dived off. Shiii. Ya aint gonna kill me"