Unless he ups the competition and does something special in the next year or two, then i think Floyd will in the long term be known for not fulfilling his potential. Like Joe Calzaghe, there will be a cloud over his legacy because of the big fights he didn't take when the public demanded them.
With fighters like Manny Pacquiao, Marquez, Morales, Cotto, Hopkins, etc all the questions of their ability have been asked and answered. They fought the best when the best were at their best. Sometimes they lost, sometimes they won but there are no questions over their legacy. Floyd will forever be tarnished with not fighting Tszyu, Cotto (in his recognised prime), Margarito, Pacquiao and at a push Hatton at 140. He could have been held in higher regard than the likes of Sugar Ray Leonard, but i think he will be recognised as equal at absolute best. That's no small achievement, but it is down to his own ego that there will forever be questions asked.
Ten years down the line people will ask whether Pacman would beat the world champion of the time and they would put together an answer. Where as with Floyd they will be asking could he have beaten the prime fighters in his own era.