They probably would have, but they wouldn't have had a holy book with a God that endorses slavery to help them support their beliefs.
You do realize the US is a secular country? Regardless, secularism is not atheism and has no necessary connection to it, so why are you bringing it up?
Bad example. Atheism did not justify eugenics. Social Darwinism did. Atheism is not Social Darwinism and has no necessary connection to it- indeed, many Social Darwinists were and are religious.
A better example would be dictators destroying religion in their populations (which is not secular, as I said before- it's anti-secular.) However, their death toll doesn't even come close to the death toll of Christianity from the Crusades, the Inquisition, colonialism, etc.
Christianity isn't "the white man's religion," but if you are a Black, American Christian with slave ancestors, Christianity was introduced to your family by the same people who were beating and raping them. That is an unfortunate fact. That's what people mean when they say that it's the white man's religion- not that it came to you through the white man, but specifically through a group of people who were committing horrible acts to your people at the same time that they were forcing Christianity on them.
As for atheism, it's not a position that was invented by white people or anyone else. There were Black atheists long before there were Black Christians. That being said, yes, atheism in America is currently associated more with white people, but again- the race isn't the point- the history is. Modern Black people aren't getting non-religious positions from slavemasters or oppressors.