People that were enslaved were not always from the working population. There were many instances in which they were noblemen men and women; educators; clergy; tradesmen; musicians, etc. It was not uncommon for Kings to throw their political enemies (noblemen) and those enemies entire families and villages into slavery if the King feared that insurrection was being plotted. On other instances noble and wealthy people in Countries would be captured and ransomed back to their families, but if the families didn't pay the kidnapped people would be sold to the slavers.
In Congo the King (ManiKongo) Mvemba a Nzinga (Christian name Afonso) allowed Portuguese Priests to enter his Kingdom to establish Catholic Churches and he allowed Portuguese traders to enter his Kingdom to sale goods to the Congolesse people. Afonso later wrote a letter to the King of Portugal advising him that noblemen and women in Kongo and even members of Afonso own families were captured. Here is Afonso's letter:
"... Moreover, Sir, in our Kingdoms there is another great inconvenience which is of little service to God, and this is that many of our people [naturaes], keenly desirous as they are of the wares and things of your Kingdoms, which are brought here by your people,
and in order to satisfy their voracious appetite, seize many of our people, freed and exempt men; and very often it happens that they kidnap even noblemen and the sons of noblemen, and our relatives, and take them to be sold to the white men who are in our Kingdoms; and for this purpose they have concealed them; and others are brought during the night so that they might not be recognized.
And as soon as they are taken by the white men they are immediately ironed and branded with fire, and when they are carried to be embarked, if they are caught by our guards’ men the whites allege that they have bought them but they cannot say from whom, so that it is our duty to do justice and to restore to the freemen their freedom, but it cannot be done if your subjects feel offended, as they claim to be... ."
http://www.casa-arts.org/cms/lib/PA01925203/Centricity/Domain/54/Letter to the King of Portugal.pdf
Then there was this Prince that captured and enslaved:
In Congo succession issues with the children of the ManiKongo often led to issues of enslavement of noble people.