I don't think there is a problem with standardized testing in a vacuum. Some people are more good at test taking than others but it's a skill that can be learned, and it's one of the easiest ways to assess a mass volume of people on a sliding scale. I don't think a score can capture the essence of a person's validity but it is a factor.
I think with the SAT for example, what hurts black and Hispanic test takers is that wealthier families can afford to invest more in test prep programs for their kids. In addition, Asian culture (for Chinese, going all the way back to Confucius days) emphasizes long hours of study by their kids in prep for exams to get into specialized high schools and university.
I have a friend (black) who is a single mom. Her daughter a few years ago was in grade 5, and lived a few blocks away from Brooklyn Tech. I asked my friend if she considered signing her kid up for summer schools that prep you for the entrance exam to the school. She said she didn't think the extra prep was necessary, and that they'll look at high schools when it's time. Now the daughter is at a decent charter high school. And I don't think the mom made a wrong choice per say, but between a relative lack of knowledge and a relative lack of interest and a relative lack of resources, and this is informed mostly by my anecdotal experience, the disparity you see in test scores along racial classifications makes sense.
Definitely agree with objective analyses that show that test taking isn't the best objective metric of evaluating a student. I don't know if there is a better way that's practical, given the millions of kids we have in this country. GPA? (But how do you account for differences between schools in terms of quality?)
And in the meantime, if I have kids, they gotta learn how to take the tests, cause the game is the game out here.