You're hiring the time and energy of a whole-ass person, with a decent car, with expenses to operate that car, and you have to account for all the time that person spends working and driving trying to get a fare without someone actually in the seat paying his way.
Honestly, for virtually any service that takes a significant portion of another person's time, the only reason it WOULDN'T be really expensive is if we attach far less value to their time than we do our own. We deeply believe that we deserve X dollars per hour for our own work, but balk at paying the same to a "mere driver".
I get and agree with your point, but I'd be shocked it the driver actually saw 50% of that after expenses and Uber's cut.