It's extremely uncommon for men and women. I remember running into a friend from middle school some years ago. He was saying we should hangout and catch up, he asked me if I smoked or drank and I said no. He jokingly said if that's the case we don't have a reason to hang out but I truthfully agreed with him.
It's kind of hard to hang out with people who indulge if you don't. A sober guy and a drunk don't usually click.
I can understand that, I think it’s because they make assumptions off the notion you guys are different in that aspect and they think you’re either judging them negatively, or they judge you negatively and assume you don’t have any fun.
Some dudes Ive talked to..when they first learn usually say something along the lines of “if you don’t drink or smoke what do you do for fun” but I know that’s not all they do, so it’s just what people make up in their minds regarding your character.
The friends I have that do it, don’t mind it cause we know we get along based off other things.