Not necessarily, but it can be if you face obstacles (Law Enforcement, Drug testing, financial problems) trying to maintain the habit. Also your social reputation can take a hit if you associate with this shyt and you got friends/family who don't condone it as well since there's still a stigma to be had for being a toker.
As long as you aren't building your entire life around it (I know dudes who smoke every day and you don't hear shyt about it from 'em as opposed to dudes who smoke half as much and wear their habit on their sleeves with merchandise, daily talks about weed and shytty music) it shouldn't be too much of a problem to start/stop/quit without any static.