Market research played a part in this. Most corporations do this to figure out where the most money will be and how to tap into it. Earlier cultural backlash from whites becomes acceptance when the art form becomes commodified and their children begin listening to it. In the process it gets watered down and homogenizes, so variety in this market is discouraged from an economic standpoint, artistic innovation is not part of the business plan. Businesses don't want to spend extra money marketing a variety of rap artists of different styles if they can just have a generic rap artist. This is why, hip-hop specifically, got overran in the 90s, they put out a great movie on this topic in the early 00s:
Pay attention to the music and the movies at the time now y'all got history.
Brown Sugar was an epilogue to a decade of commercializing hip-hop. You can go back to the 80s, but that's really an introductory period.
The 90s was the golden age, and businesses were trying to figure out what variety of hip-hop would stand the test of time and be more marketable after the millennium. The conscious rapper won't survive the media (radio) if the clubs and radio are in agreement. It gets pushed further underground, if artists get signed or were signed they get a reduced budget for their albums, flop, get dropped from the label, pivot to what's selling or burn out. There was a slower time marrying hip-hop and R&B, the early 00s is when it really took off in the mainstream.
Violence in hip-hop became more marketable in the clubs, either through gangsta rap or content.
It Was Written is a great example of the concept that shifted black music, it established the standard that was further reworked for artists like Ja Rule, 50 Cent. Tupac played a part in how a rapper should look for a period of time.
There were "conscious" artists and rappers still with artistic integrity but they had to adjust to the overall changes in the mainstream in one way or another. Some were more successful than others. You can look into Common's career at the end of the 90s and
Like Water for Chocolate, Soulquarians was the last bit of consciousness hip-hop had associated with the early 90s and that was Questlove and Q-Tip (The Roots and Tribe had albums out before '94). Common was out in '92, but he wasn't an East Coast rapper. Most of them Soulquarians migrated to New York.
A lot of research went into studying black communities in New York, Harlem and Brooklyn to be specific and had to do with the riots in the 1960s.
and the biggest thing, young black folks politically acting like old conservative racist cacs in a attempt to be edgy and shunning anything “black” even each other 
I know some of y’all think it’s “just music” but it’s deeper than that. Soulless music is just a early symptom of a bigger issue for us, just like erectile dysfunction is just a early check engine light for heart disease, we are dying. African Americans are the soul of this earth and so many of us can’t see the forest for the trees 
da 90’s what is the pinnacle hip-hop, R&B and all urban music ….. da hell you talking about 
