Like other posters have said, SEGA'S crappy business decisions in the 90s ultimately caught up with them.
* Consumers stopped trusting Sega after dropping unnecessary back-to-back addons for the Genesis that had very few worthwhile games (Sega CD, the 32X).
* SEGA launched the Saturn way too early in the West. (They didn't listen to Tom Kalinske)
* Retailers weren't ready to advertise or stock Sega Saturn units, and third parties weren't informed ahead of time so they could make any games for the Saturn.
* Most of the Saturn's best games (including 2D fighting games and RPG's) were never localized in the West because of Bernie Stolar.
"Bernie Stolar dropped the ball with the RAM cartridge/X-men Vs. Streetfighter could've expanded the market."
* They weren't taking the best care of their mascot. They struck gold when they came up with Sonic, but since SEGA's American and Japanese branches weren't getting along, they couldn't finish Sonic X-treme. If the OG Sonic Adventure had been a Saturn game, they would have really done some damage.
Meanwhile you had Mario 64 and Ocarina of time crushing the buildings. And to be fair, Nintendo made a lot of mistakes in this era. The Phillips CD-i and the Virtual Boy were both trash, and Nintendo alienated 3rd parties (esp. Squaresoft) by choosing the cartridge format over CD's for the N64.
...but at least Nintendo consistently put respect on their mascot. (Plus they had the handheld market on lock, especially after Pokemon changed the game.)
* Sony saw that Sega didn't have it together and took advantage by launching the Playstation with a better game lineup and a lower price.
(That's when Sony ethered Sega with the "$299" comment in '95).
The saddest part is that SEGA came correct after all that.
The Dreamcast was an awesome console with plenty of great games. (Jet Set Radio, Sonic Adventure 1 and 2, Crazy Taxi, NBA 2K, Shenmue, the Capcom fighting game ports, etc)
...but the DC died early because of the unstoppable momentum of the Playstation 2 coupled with Sega's stained reputation from their bad decisions in the mid 90s. If the Dreamcast had a DVD drive, it could have been way more of a threat.
It didn't help that the Dreamcast had weak copy protection, making it way too easy for people to bootleg their games.
Looking back, it's pretty crazy how Sony gained such a huge advantage from both Nintendo and Sega not having their act together during this era. And I'm saying that as Nintendo fan to the grave.