People did not consider the perspective of Stephen.
Candyland was located in the cotton kingdom of Mississippi.
And the state of Mississippi practiced the most heinous forms of slavery out of all the south.
In the most brutal place to be a slave, here we find a black man, wielding relative power and privilege, traditionally reserved for white men, within the boundaries of Candyland.
Stephen was its true master and exercised real authority, over whites and blacks, on that plantation. An unheard of position to find a black man in Mississippi.
Until.....
Some uppity nikka on a horse not from the area comes into Candyland and threatens to disrupt all that energy.
Stephen, as the true authority over the plantation, had a duty to protect it. Because outside of it's walls, he was just another nikka. If it went down, so did he.
He had to ensure that Candyland was prosperous and that Calvin, the instrument to which he exercised control, was protected.
As long as both were good, his unheard of position could be maintained.
Django and the German threatened the stability of Candyland with their deceitfulness and therefore Stephens position, which was the catalyst for him acting in opposition. He was only trying to protect his investment and his position from an outsider who would have came in, and, after disrupting its energy, left with Stephen having to fix things in his wake.