Because he robbed her of her ultimate goal in life
That 14 shyt is meaningless in the world they live in, especially given Ellie's background who went through way too much shyt to be treated as a kid that just entered highschool in a normal world.
This shyt is not about letting her decide whether she can go to a sleepover or not, this is literally her purpose as she explicitly states it repeatedly throughout the series and at the end when she recalls what happened to Riley.
Btw, Joel recognizes her agency, 14 be damned, in episode 6 when he steps up to her in Jackson and says "I thought about it, and you should have a choice...". This is one of the set-ups that I don't remember being in the game, and that I liked in the show.
Please don't mind my argumentative tone, nothing personal, I just missed those part 1's ending passionate discussions