As XKCD so eloquently put it, a person who falls back on "freedom of speech" to support his argument has made the ultimate concession. He has admitted that the most meritorious thing he could say about his point is that it literally shouldn't be illegal.
I'm a big fan of freedom of speech and the whole "market of ideas" analogy. I really am. I would even consider myself a civil libertarian.
But we're living in a United States where the president, his administration, and his party are not interested in participating in a market of ideas. They're interested in agenda-driven "alternative facts." Breitbart editors, like Milo, are not interested in having a debate. They're interested in provoking an angry mob, either to support their ridiculous agenda, or to oppose them so that they can cry foul and pretend to be victims. If you really want to take the "market of ideas" analogy further, these sorts of people are the equivalent of antitrust violators in the market of free speech.
If people aren't interested in facts, then I have no sympathy for them when their speaking engagements get protested or when those protests get a bit heated. None whatsoever. Yes, violence is generally bad and I don't condone it. Yes, freedom of speech is good and I generally condone it. But I feel no compulsion whatsoever to stand up for Milo and his nonsense.
If you want to abuse your right to free speech by spouting nonsense, don't be surprised when people exercise their right of free speech to shout you down and shut you up.