Unfortunately for OP this is correct. The fact that his brother is carrying on like this means there is both a lack of respect and a lack of education. White people in general don't have empathy in these situations because: 1) they benefit massively from the status quo, 2) they don't want to be guilted or shamed bc it was "a long time ago" (though shyt still goes on today and is actually documented), 3) they don't know what they can do about it, 4) black people don't really know how to present their case or are easily triggered, 5) things have always been this way, 6) there are unresolved feelings of both superiority and inferiority, 7) lack of education and how it's shaped the world we live in, 8) it's just jokes in their eyes and they have no understanding of how they are already being shaped, amd so on.
But in the end OP, if you want to try and get through you will have to try and get dude to empathize with you as a person first, then as a black person, then as a black person interacting with a white person, then as black person interacting with a white person who happen to be related.