If that was the case, more people would do it. There is a major disconnect in these higher maths. If you make it a requirement, only 30% of people would graduate high school and would have suicide rates of Japanese students. There would be a major backlash in this.
As I said, the reason why more people don't do it is because they look at how complicated it is from ALL of the simple steps put together. Most people do not have the attention span or the focus to solve a problem involving 50 different steps, even if the individual steps are pretty easy. And I know this for a fact because most of calculus is just a bunch of algebra and algebra is not hard.
The good thing is, I believe most people have the ability to improve their focus and work ethic. It just takes time, like everything else. There is nothing stopping the average person from succeeding in calculus except for their own lack of focus.