I've heard this as well. There does seem to be some truth to this on a genetic level at least.
Yh it’s kind of obvious when you look at it. Pre-colonial Akan culture was different from most lower west African groups like the fon, Yoruba. Etc
This is an excerpt from Al bakri on ancient Ghana dated 1067
“This is their custom and their habit, that the
kingship is inherited only by the son of the king’s sister. He has no doubt that his successor is a son of his sister, while he is not certain that his son is in fact his own, and he is not convinced of the genuineness of his relationship to him.”
That’s matrilineal kinship. Akans are the only matrilineal group in lower west Africa.
But the culture of the Sahel has changed a lot now since Islamic influence. Our culture was more similar to the pre Islamic sahelian culture.