The TLDR answer is that it comes from the 80/90's NYC hip-hop scene, and ultimately "nygga", became a hip-hop term. Some native NYC'ers are a bit naive about how segregated the rest of the world is, so they don't realize how offensive it is.
It goes back to 70's & 80's where Blacks & Latinos (I'd say mostly Puerto Ricans) were living side by side in the some of the same NYC projects together. Hip-hop help bring those two groups together in such a way that a more intimate bond was created. You had a lot of Puerto Rican b-boys, b-girls, graf artists, DJs and even some rappers back then. Those particular Latinos were living the same inner city struggle, and had a lot of shared experiences. Close to half of PR's in Puerto Rico are reported as having "significant black West African ancestry". Many of them don't like to acknowledge it, but the ones that were born in NYC and were into hip-hop tended to embrace it more. So the youth adopted the slang, and on some level they felt like they were "nyggas" too. It became part of the hip-hop experience, especially in certain areas of Harlem & the Bronx. Even before hip-hop got big, young Puerto Ricans were already starting to differentiate their experience from Puerto Ricans on the island, hence the term "Nuyorican".
A lot of popular rap artists from Harlem & the Bronx are part Latino. A few infamous street figures that are associated with hip-hop are Latino. If you were getting busy on the mic, getting busy in the streets, not scared to attend pre-dominantly Black social events then you pretty much had that pass. As hip-hop started to blow up the usage of the word became more widespread, and many NYC Latinos were using the term, mostly unchecked. A lot of Black people didn't like it, but didn't have the time & energy to stop groups of Latino kids from using the word. Some of those Latinos don't necessarily identify with Black people at all, but they grew up using the word.
The thing that a lot of inner city NYC kids didn't/don't realize, is that Blacks & Latinos didn't have that type of bond ANYWHERE else on the planet. There's a funny story Fat Joe told about how he brought Kid Frost (Mexican Cali rapper) to his hood, and Frost was legit confused and upset about Joe bringing him around so many Black people. Obviously we've seen Cortez catch backlash over using the word.
That takes us to guys like Money Bagz. He's not Black, or Latino. But he's an NYC hip-hop kid that was dibbling and dabbling a little in the streetlife, whether it was gangs, hustling or whatever. He self identified as a "nygga" as well. There was an assumption that he was Latino at first with him being around all Blacks and Latinos, so I don't think people were coming at him. He got cool with a lot of people, and by the time everyone was like "Yo hold up, he's 100% WHITE" he was already "down". You know how male bonding works. Maybe a Black dude tried to check him, put hands on him and he fought back. He comes back around the next day, it's "nah he cool". Or you (as a Black man) get into a situation, and this dude is fighting right beside you. He was willing to shed blood for you, that's your "nygga" now. We've heard Diz use the term, and apparently none of those Grape Street Watts Crips that he hangs out with check him on it. I guess his situation is somewhat similar, but he avoids using the word in his raps.
I'm not saying this any of this is OK. I'm just giving my take to help give a better understanding.