You can't really pinpoint it to one thing - Kanye's career was one perfectly executed move after another from 2003-2010.
College Dropout was a huge success for what it was - it bucked the trend started by 50 Cent and did well on its own merits, even if it's pretty rookie by his standards today. LR had his first megahit, but it also succeeded in bringing in a much wider critical audience - you're looking at AOR and indie fans when you've got Adam Levine and Jon Brion (why he didn't stick with him for longer I'll never know). Graduation is still one of the best festival albums I've ever heard. The fact that he could make an album like 808s and not receive a critical mauling demonstrates how well Def Jam/Rocafella pulled off Kanye's career, and MBDTF/WTT just solidified all of it.
The thing that you need to note is that each album appealed to a particular fanbase - the reason Kanye is a megastar now is because he put in that work building up a fanbase over years. To be honest, I liked College Dropout but I wasn't sold on Kanye until LR, which remains my favourite. Some of my friends who love going to festivals love Graduation, but I don't think they've ever heard the first two albums. But I can guarantee that they've all heard the stuff he did since then.