For the fantasy medium, I dislike race swaps of this nature too, but its not because of source material accuracy or the sort of complaints I see the anti-sjw crowd make. I dislike them because it misses out on the opportunity of creating and exploring a fantasized black people with their own culture/identity that takes inspiration from real world black/african cultures.
For instance, because Asians, Arabs, and Europeans are depicted in most fantasy to be analogs of their real world cultural counterparts, they end up getting cool stylized takes on their historical real world warrior classes like "fantasized" knights, barbarians, vikings, romans, samurai, arabian knights, ninja, and shinobi. That's the type of representation I like to see black people receive rather than just purely being European knights with a European aesthetics/culture. Its just so boring to bring in non-whites and have them ropleplay as medieval brits rather than giving them their own culture and then show how all those different cultures interact with each other.
A good example of this occurring in diverse fantasy universes with multiple races is something like Redguards from the Elder scrolls series, they are a black race that is a fantasy analogue of the West African Mali kingdom/Moors. Or, even the summer islanders from the Game of Thrones universe. Though they didn't get much attention in the show, they are a black people that act as analogs to various African and afro Caribbean cultures. You see, that's interesting.