I actually know what you're talking about and I feel where you're coming from.
I've been considered a good looking dude by women who's in shape, but the majority of my life was spent as a token nerdy black dude who lived in a place (Pittsburgh) where he was the outsider so that kind of killed my self-esteem with women. When you don't "fit in" a person is going to be really hesistant to go out there and get basic things done that everybody else can do easily. You're going to be hesistant to ask for work. You're going to be hesistant to meet women. You're going to be hesistant to ask for a promotion. For me those problems were always finding work and meeting women. Especially going out with groups of friends where women would flock to them and I would be alone at the end of the night. SO, I just really stopped pursuing women for a good period of my life in college and just kept to myself and my close friends. I just found it difficult and ended up being alone/sexless for years at a time and I just swirled into a severe period of depression. It really made me hate guys who had no problems meeting women with a violent passion for a while and here I am at almost 30, in better shape mentally and physically and still trying with those feelings of resentment because of a past I couldn't control and feeling like I'm just relearning how to interact with the opposite sex.. I didn't want to be that person, I had moment of my life in my early 20s where I could meet women easily..but as time went on and I realized out of place I was where I lived and how I didn't like where my life was at, I just shutdown. Now I'm in a better spot than before...being broke is what makes me lack confidence in meeting women now. I'm just sick of this society and it's mores, games, rituals, rules, and BS to the point where I'd rather not even care about meeting women. My life isn't driven by sex anyways.