It looks like Ginsburg admitted to smoking pot, even though that was an absolutely ridiculous reason to withdraw. But given how long Kennedy was on the court I'd have to assume this was back when weed wasn't looked at (well, for white people at least) as that "harmless, cool drug."
The difference is Kavanaugh has been accused of sexual assault and attempted rape 36 years ago by one woman and until she testifies we won't be able to see how credible that accusation is. There's already a talking point that she doesn't remember pertinent facts about the incident so until testimony changes that it would look like an outright admission of guilt for Kavanaugh to withdraw his nomination. You also have the precedent that a mere allegation by one person can derail a supreme court nomination so I'm sure it's a tactic that would be used again if he withdrew before the accuser actually testified.
I'm not saying he did or didn't do it and he's not on trial so there doesn't need to be proven beyond reasonable doubt, just given enough credibility where it looks possible. The one major thing I see Republicans doing to shoot themselves in the foot is trying to downplay the accusation (it wasn't successful so it's okay etc). They should really keep their mouths shut on this one and let the process play out. Kavanaugh is such a limp noodle I guess they feel the need to defend him but the Dems have staked everything on this so if her story withered under scrutiny the GOP would have won the day without having to mount a defense. Now, they put themselves in a position where even if it is a less than credible accusation the whole handling of the matter will be an indictment on the party. But with the 53% does it even matter? Sometimes I think the media overstates outrage and concerns over these issues.