Yeah, but some modern libertarians count Black people as individuals while also presenting themselves as strict constitutionalists when it comes to government reach, whereas the framers of that very same constitution obviously did not consider Black people such. If libertarians were around during slavery, they wouldn't necessarily have legally opposed it, assuming their views on government were the same as they are now, for the simple fact that it would constitute drastic and serious government intervention into the status quo. Also factor in that slavery was contentious from the very beginning of the constitution's emergence- on those issues, traditionally, libertarians have opposed using the government to silence a close minority or majority on civil rights issues or on things like gay rights (not just marriage, but basic privileges of civil unions, for example,) so we have no reason to believe they wouldn't have done the same with regard to slavery, adopting either a states rights stance, saying that it would take care of itself through the market and without legislation, or even saying that it didn't need any regulation or change at all.
You yourself might not hold that position, but there is a strong tension in libertarian philosophy that makes it a coherent position to hold, and is why many people continue to hold similar positions (for another example, the position that no public anti-discrimination laws are necessary, allowing the majority to have de facto power over minorities, while simultaneously decrying democracy itself as majoritarian evil.)