When you post, I'm interested in your answer.
I know that with white men, a lot of the time it has to do with a superiority complex that comes with racism. They see someone that they see as "undeserving" of a white woman because by having darker skin, you're not worthy to date "their" women.
It can feel like it sucks sometimes to be thought of as "inferior," but being the quintessential optimist, I try to see the silver lining in this. If I'm given the motivation to do really well in life because people see me as someone that really has to prove themselves to be seen as worthy to date anyone in the gene pool, I'll take all the motivational fuel that I can get. There's something particularly satisfying about receiving the begrudging respect of someone who would normally go out of their way to think less of you.
Overcoming that lower part of the brain that motivates hatred and prejudice is hard, so when it gets done it's incredibly satisfying.