Honestly, for me, it's mostly how obnoxious the Olicity fans are that has caused people to overtly hate the pairing/ship. This explanation is in no way meant to be rude towards, but I do have a tendency to phrase things differently than I intend to, so apologies in advance.
Here's a list of some of the things that have turned a lot of people off:
The Twitter rampage. All in all, the big Olicity movement has been done by a couple thousand posters who constantly post the #olicity tag on everything. Everybody gets that Olicity is a fan favorite ship. Just because it consumes everything about why they watch the show, doesn't mean they have to advertise it on everything. Stephen Amell could Tweet: "Working in the foundry today and it's freezing"@amellywood. And there would be hundreds of tweets later saying "Huddle up with Emily, she'll keep you warm #olicity" or "What's the foundry? Does that mean #olicity?" For most of the fans who watch the show and don't care about who ends up with who, it's a major turn off when all they see are people propping up a ship that hasn't happened yet.
Which leads us into the Facebook assault. Stephen Amell is an amazing actor. While he probably won't be winning an Oscar or Academy award anytime soon, the level of interaction he has with fans is amazing and unparalleled. He shares so much of his life and work with people that fans have truly fallen in love with him. Unfortunately, there's about 20 people who literally post the same photo to every comment, asking for Olicity. And another 10-15 fans who will go on there and just type #olicity in the comments. To every post. Whether it's a training video, Q&A, or just a status update. Which, again, is a major turn off for people who are watching the show to see Green Arrow.
For a lot of people, having it shoved into our faces constantly like that, is enough to push fans away from the pairing and the show. Even though they like Felicity, it can turn people cynical towards the character and the ship.
There's also the fact, that in the comics Black Canary and Green Arrow get married and Felicity Smoak marries Ed Raymond, the father of Ronnie Raymond, who's half of Firestorm (and being played by Robbie Amell this fall on Flash). Nobody wants the show to follow the comics verbatim, but there are quite a few fans who want them to stick to key points of the comics (who marries who, for example). (For the record, I personally don't care if Oliver gets married in Arrow)
Now let's get into the truly juicy stuff.
There are three kinds of fans who like the pairing: The ones who just think they would be cute together, the ones who think they should be together because of "x" reason (the shipper) and the ones who have dedicated themselves to force it to happen (the hardcore shipper).
The first two aren't that bad. I don't have a problem, personally, with people who like olicity in general. It's the third kind that has destroyed peoples perceptions of olicity.
The ones who stampede Twitter and Facebook with #Olicity and demanding that Oliver get together Felicity. There also the ones who cry foul whenever Stephen Amell posts a picture on facebook or twitter with a cast member who isn't Emily Rickard.
The ones that posts hatred towards the cast members because they're interfering with Oliver being Felicity. If you spy on those lunatics, you'll see that whenever Stephen or the Producers posts something about Felicity, it's all kittens and rainbows. If the producers post something that might interfere with their ship (Ray Palmer or Laurel becoming BC for example), it's all filled with hatred and demanding that the producers/Stephen Amell change it to make it focus on Felicity. Stephen recently posted a picture of him and Katie together that Paul Blackthorne took of them. The comments ranged from, "What a nice picture. Paul did a good job" to "This picture is terrible, why do you have Katie there instead of Emily? It should be Emily, take it down. #Olicity"
One thing that most of them agree on is that they hate Laurel/Katie Cassidy. Katie Cassidy is the most abused actress on that show. The amount of hate that spews from that dark corner of the fandom is absurd. They have morphed their perception to going from hating her because she and Oliver are close, to hating her because she's done x, y and z. Pick any minor thing that Laurel/Katie has done and they've called for her to be killed off or written out. They've even threatened Katie, personally not the character, demanding that she quit the show so olicity could happen.
And it's not just Laurel that they hate. When Oliver and Sara were together, the same thing happened to Caity Lotz. Same thing with Helena. Hell, I'm sure if Oliver and Diggle developed a romance, they'd even hate Diggle/David Ramsay, who's the Firstmate of their ship.
Which brings me to my final item on the list. Their Grand Delusion. Tumblr is a cesspool of fan fiction, which is fine and dandy because that's what it's for, fan fiction. The problem is, a few folks over there have taken to making memes about the show, portraying olicity moments that didn't actually happen. Scenes spliced together to tell a story. Actors with quotes over their head that were never said in the show. Mostly harmless stuff that doesn't matter. Except when you get people who don't actually watch the show and merely observe to see the Oliver/Felicity pairing. Then these fan fiction memes and stories become the real thing. All of a sudden, the entire show is nothing more than the Smoak and Arrow show. All of the other characters exist to promote this romance. Diggle is Firstmate, who is their chief shipper. Amell is the captain of the ship because everything he does and says suddenly points to olicity. Laurel is evil because she takes Oliver away from Felicity. I could go on, but I won't.
These things don't stay on tumblr though, they migrate to other media. They honestly believe that the fan fiction they saw there is what actually happened and they come on to other sites promoting it and then wonder why it is that people don't see what they see.
A rather lengthy post to answer a relatively short question, but it's a complicated question that deserved more than, "Olicity is bad, mmkay." And there's more that I could expand upon, but I won't. There's a lot of people on this board who like Felicity but hate olicity. And I'm sure most of them will agree with what I said, but obviously I don't speak for everyone. And I'm sure there will be some from the olicity crowd here that will say "nah-uh, it's not like that". But it is.
In conclusion, it's not so much that people don't like Oliver and Felicity, it's that they don't like the fans of the ship, and don't want to see it on the show.