While that is a good point it is not good enough to justify reimbursing people that took away the freedom of other humans, forced them to work suicidally hard without payment, tortured them, raped them, and then repeated the process with their children.
The crimes against humanity of the Nazis or of the Imperial Japanese or the "crimes" the French royalty were killed for weren't illegal at the time of them being commited either. In legal principle that means that they shouldn't have been punished, yet you'd be hard pressed to find someone who would support that.
Furthermore, freeing the slaves wasn't even a punishment to the slavers, it was simply (a step towards) the restoration of the natural order and rule of law to the equality of men.