How do you distingush by phenotype? What makes a redheaded, freckled European the same white as a blonde, blue eye Swede?
societal/cultural norms
There is no objective reason for racial categories to exist for the exact reason you pointed out. Most people in racial categories don't really look that much alike. The Yoruba, for example, don't look like the Maasai, but both are considered black. Swedes don't look like Italians, but both are considered white. Japs don't look like Indonesians, but both are considered asian.
Race (as we know it) only exists because a few hundreds years ago some Europeans decided to make it. Before that, race (as we know it) didn't exist. People just considered those of the same tribe/clan/ethnic group their race and everyone else was the other. No African ever considered everyone with black skin the same race. Similarly, no European thought everyone with white skin was the same race. The Ancient Greeks were notable for not considering themselves the same race as the northern Germanic tribes.
Race is simply a social construct. Nothing more. And there is no OBJECTIVE justification for it.