I think it's the way and context of how it is used at times.
Some guys use it in a way as if they're talking to or about a different species all together.
It's kind of like how the word Black isn't offensive in of itself but, some people will use it in a dehumanizing, objective way: "Those Black people!", "You Black people", or "The Blacks" (something about that last one really rubs me the wrong way).
Sometimes a word has enough history behind it to make it unacceptable regardless of how it's used (Like the n-word). However, a seemingly innocent word can offend the masses and destroy one's credibility just in how it's used.
Regardless of the word, people seem to forget that there can be a huge difference in response just by one's tone, infliction, body language, and context of a word or phrase.