I don't even think it's a matter of people not caring, it's a matter of those reference tracks not showing that Drake isn't responsible for writing most of his shyt. Reference tracks are not a new thing in hip hop by any means. Sometimes writers get writers block and need someone to come in with ideas to get the creative juices flowing. Some of your favorite rappers have done the same thing, but you'll never hear about it.
There's also the tricky issues of Drake being a writer for other people and also the matter of how personal his album cuts are. It's hard to consider someone a fraud when it's a documented fact that he's pinned hits for other people.