Every female who went out of her way to tell me I was ugly, was ugly herself. So the Kettle calling the pot Black. My response was to not care what their opinion was on how attractive I am simply because I was not attracted to them at all. So, what I am saying to you is don't worry about what a woman says about your looks when you have no desire or opportunity to pursue her.
Also, always being down on yourself and showing self-pity is not attractive to any well-adjusted woman. You can be realistic about how good-looking you are without harping on it constantly. I have never been a pretty boy, but I have no problem getting choosing signals at 53 from women ages 25 - 65. I decided as a late teen I would reduce my negatives down to one thing, my ugly mug. For the rest of my attributes, I worked to make them positive. You can do the same thing, my brother.