What doesn't sound right about it? People obviously migrated in waves over a period of time, not only that but some people continued living in the desert for thousands of years after the encroachment of the desert. The people in the Sudan and in Egypt still live in the desert to this very day.
The Nok culture in Nigeria got to that region about 3,000 years ago.
http://en.wikipedia.org/wiki/Nok_culture
The Wolof people of modern day Senegal didn't get there unitl 640 A.D.
http://en.wikipedia.org/wiki/Wolof_people
Pottery from 2500 BC was found in modern day Igbo States in Nigeria .
http://en.wikipedia.org/wiki/Igbo_people
So clearly people did not leave all at once, but it you read the write up on the Wolof people you can see that they settled in Senegal due to the encroaching desert.
Another thing that is clear is that people did not go directly into the forest belt of West Africa. That did not start happening in any great degree until Africans started to smelt iron. Iron was made into weapons and tools for farming and clearing the forest, which is what allowed further pushes into the forest belt (aka jungle).