Statistics vs. storytelling: the grudge match?

[ogc163]

Narrative journalism has been dogged for years by the idea that it is too subjective or somehow less capable of conveying hard numbers to the public than a traditional news story. In a world where data mining and visualizations have become more fluid and accessible, it’s no surprise that the tension between numbers and narrative has not disappeared.

Yesterday, I was talking with an MIT new media researcher who showed me a project that incorporated individual narratives. He felt the need to reassure me that they were not just stories but “stories with teeth.” Dan Conover over at Xark (see item 4 in this post) has for a while now been calling for an end to what he refers to as “narrative-based” news. And Los Angeles Times education reporters caught minor flak in September on Numbers Rule Your World, a statistics blog that has taken to shouting “Story time!” when they think a reporter has jumped the shark, going from data to inventing explanations or causes.

Late last month, John Allen Paulos of Temple University wrote on “Stories vs. statistics” for The New York Times’ philosophy blog, referencing the inevitable C.P. Snow’s Two Cultures lecture and considering how the literature/science divide plays out with regard to what kind of information we prefer. Outlining the tension between stories and hard numbers, Paulos summarized two kinds of errors in statistics: “we’re said to commit a Type I error when we observe something that is not really there and a Type II error when we fail to observe something that is there.” A bit later he writes that which error we worry most about may determine how we want to receive information:

People who love to be entertained and beguiled or who particularly wish to avoid making a Type II error might be more apt to prefer stories to statistics. Those who don’t particularly like being entertained or beguiled or who fear the prospect of making a Type I error might be more apt to prefer statistics to stories. The distinction is not unrelated to that between those (61.389% of us) who view numbers in a story as providing rhetorical decoration and those who view them as providing clarifying information.

Narrative fraud

As early as 2007, Nassim Taleb noted “the narrative fallacy” in reasoning, which he suggested might be better titled narrative fraud. While not knocking narrative nonfiction directly, Taleb indicted our human tendency to link events by creating stories that explain them, the comfort we derive from thinking we understand why something happened if we make a story out of it.

In a TEDx talk from last year, which we covered on Storyboard, economist Tyler Cowen encouraged listeners to distrust stories in proportion to the very degree they find themselves moved by those stories, to understand that when a story begins to convince them of something, a sales job is underway. There are only a few story templates, he noted, and stories tend to leave out conflicting or messy stuff that doesn’t fit the pattern. “The more inspired a story makes me feel,” said Cowen, “very often, the more nervous I get.”

I have no issue with Cowen teaching us to be more critical consumers of story or to ask what information is being left out. And at root, it’s true that stories can be reduced to a few basic themes. If you’ve read one dramatic medical narrative in a newspaper, it’s not hard to find a dozen similar ones on the same topic in other papers that use more or less the same machinery. At times they can be reductive, generic or manipulative. But sometimes something else is happening.

A narrative like Los Angeles Times reporter Thomas Curwen’s “Ana’s Story” might on a surface level have a lot in common with Tom Hallman’s Pulitzer Prize winner “The Boy Behind the Mask.” Both are about coming of age with a facial deformity and undergoing harrowing medical procedures to address the problem. But the two stories’ subtler aspects are more complicated and different than their common template would lead you to believe. Looking at the broad story, it’s easy to miss the specifics that can matter to a given community at a specific point in time. Curwen informed me that when his piece on Ana came out, it was the second most viewed story for the Metro section in the paper’s history.

Not all narratives are the same

People have a tendency to create their own narratives out of stock stories, which includes adding their own meaning to the stories they view and read. It’s worth realizing that those meanings aren’t automatically irrelevant or wrong.

But even if we concede the existence of a buffet of empty-calorie narratives, highly-engineered true stories that engage readers but don’t offer the kind of information that might appear in breaking news reports or policy stories, it doesn’t mean that narrative can’t be useful in conveying news. One celebrated example is This American Life’s “The Giant Pool of Money” episode. Jay Rosen did a good job of explaining why such a narrative was important—the idea that without background on the financial collapse (and, I would argue, recognizable characters to follow through it), many of us were unable to understand individual hard news updates about the economy. Narrative journalism can provide context in ways that not only educate people but that they also value (check out Nieman Lab’s post from this summer about how Slate racks up visits with long-form journalism.)

Narrative as a vehicle

The most important thing, however, may be that narrative appears to be the most efficient vehicle for getting people to understand, remember, or accept new information. Studies done within journalism and outside it have confirmed the power of narrative in conveying information. Previously on Storyboard, we also noted a Kaiser Family Foundation study that looked at the real-world effectiveness that even fictional television characters and narratives can have with regard to conveying public health information.

I recently interviewed Safra Center fellow Michael Jones, who is melding the worlds of statistics and story by analyzing storytelling’s effectiveness in policy arenas. (For his purposes, he identified a story as anything that involves characters, a plot, a setting and a moral to the story or a solution to the problem.) Jones noted that when it comes to policy, not surprisingly, that “even if the narrative is incomplete in a news story, people will fill in the blanks with what they already bring to the table.”

He later added that “[t]he biggest obstacle is believing that you have to communicate narratively to begin with, as opposed to just conveying scientific information to people and letting them fill in the blanks. You have to tell people a story.” As for worries that structuring a story for maximum impact is somehow cheating, Jones noted that most reporters probably want people to read and understand their stories, and suggests that having tools to do that more effectively is not a threat.

If statistics and studies help us better understand how narrative works, and reveal the mechanics by which narrative techniques persuade readers, pretending that we don’t have that information isn’t an option — though that knowledge may add urgency to the journalistic questions of objectivity, transparency and advocacy. And wishing that people didn’t have a tendency to want things tied together in stories won’t make it so.

Story partnered with statistics

It’s worth remembering that researchers and journalists alike are capable of overstating their conclusions or misinterpreting why something happened. And numbers fans dismayed by stories have their own critics: scientist-turned-filmmaker Randy Olson has been trying for years to get scientists to understand the value of storytelling in sharing research.

Both researchers’ and journalists’ work involves focusing on a limited topic, investigating it, and then presenting information about it in ways that can be understood by target audiences. This doesn’t mean that narratives have to pander or oversimplify, though different audiences will have different thresholds for how general or specific information can be.

Simple tools like hyperlinks have made it possible to effectively embed data in narratives and allow readers to evaluate source information for themselves. Comments and social media outlets have given readers options for giving critical feedback that more clearly defines the story. And with tools like IBM’s Many Eyes and Gapminder, which now allow data visualization to combine some strengths of stories with hard numbers, the distance between story and statistics may not be such a vast divide. Even Nassim Taleb’s book criticizing our worst narrative impulses is filled with instructive stories that illustrate his points, make his arguments and entertain the reader.

Statistics vs. storytelling: the grudge match? | Nieman Foundation

An old article, but one that is relevant given the current circumstances in regards to politics and the coronavirus. And it helps explain why people have difficulty thinking in probabilities and/or counterintuitively vs thinking in narratives. But that is not to imply that narratives are generally a bad thing, as I think that the pop behavioral econ/psychology goes a bit too far in their criticism of intuition. Instead, persuasive narratives and statistics should be combined to accurately convey information. Thus, providing people with correct info while also keeping them from being bored.

 

[ogc163]

 

[Cadillac]

this seems like a interesting read, in for me to read it later.

 

[F K]

I don't really understand the battle here. How do you explain the impact that statistics have without telling a narrative story of some kind? and things that are statistically improbable happen all the time. Should we avoid telling true stories that run counter to the statistics because they are unlikely?.

Not to mention that statistics for the most part work by explaining why and how things happened in the past, which means that they are not enough to create a future path with.

 

[ogc163]

I don't really understand the battle here. How do you explain the impact that statistics have without telling a narrative story of some kind? and things that are statistically improbable happen all the time. Should we avoid telling true stories that run counter to the statistics because they are unlikely?.

Not to mention that statistics for the most part work by explaining why and how things happened in the past, which means that they are not enough to create a future path with.

1) We should avoid telling narratives that fit with preconceived notions AFTER the situation has unfolded, which can often lead to us not taking complexity into consideration. And it would be beneficial if we separated outcomes that are highly dependent on luck and randomness and are dependent on a wide set of variables vs. outcomes that scale and replicate and depend on a small set of variables, narratives often get in the way of this separation. Thus, when it comes time to reflect and learn from past decisions and outcomes it's better to focus on things that scale.

An example of this is when Black white-collar professionals who grew up poor are pushed as examples of being able to persevere through adversity, and yet when you look at their past often times there are high impact variables that do not scale or have low transferability. And their narratives are pushed to showcase an ability to persevere through a difficult journey, but if people thought more probabilistically they would take into consideration concepts like outliers, frequency, scalability, and compound events. But that doesn't mean we should not tell those stories, but we should acknowledge the limits of how much insight those stories actually provide.

2) I disagree with the notion that statistics cannot tell you what will happen in the future, as its utility depends primarily on the domain in which it is being used. Where there are a lot of variables and a large time horizon (i.e. Stock Market) it is not as useful as where the variables are relatively small (Blackjack).

 

[F K]

1) We should avoid telling narratives that fit with preconceived notions AFTER the situation has unfolded, which can often lead to us not taking complexity into consideration. And it would be beneficial if we separated outcomes that are highly dependent on luck and randomness and are dependent on a wide set of variables vs. outcomes that scale and replicate and depend on a small set of variables, narratives often get in the way of this separation. Thus, when it comes time to reflect and learn from past decisions and outcomes it's better to focus on things that scale.

An example of this is when Black white-collar professionals who grew up poor are pushed as examples of being able to persevere through adversity, and yet when you look at their past often times there are high impact variables that do not scale or have low transferability. And their narratives are pushed to showcase an ability to persevere through a difficult journey, but if people thought more probabilistically they would take into consideration concepts like outliers, frequency, scalability, and compound events. But that doesn't mean we should not tell those stories, but we should acknowledge the limits of how much insight those stories actually provide.

2) I disagree with the notion that statistics cannot tell you what will happen in the future, as its utility depends primarily on the domain in which it is being used. Where there are a lot of variables and a large time horizon (i.e. Stock Market) it is not as useful as where the variables are relatively small (Blackjack).

1) I kind of disagree with that too. We should probably just have some additional section that includes how likely the outcome in the story was, or maybe add a "most common path" narrative to counterbalance, but our fates aren't predestined, reading an article about a statistically unlikely path out of poverty might help someone else take that same step.

2) We are in agreement. I basically filtered out stuff like sports and gambling , and text prediction because they have such small variables but that wasn't fair. I just think that people overhype the predictive power of statistics on complex problems.

 

[ogc163]

Stories are bad for your intelligence​

How Historians (and Others) Make Themselves Stupid​

This is Henry Cort. You probably haven’t heard of him unless you’re an Industrial Revolution nerd (I hadn’t until this week). He isn’t as well known as James Watt or Joseph Priestley - he wasn’t one of the Lunar Men - but Cort played a critical part in the creation of the modern world. He invented a method of production which made it much easier and cheaper to turn scrap iron into high-quality iron, ready to build railways, warships, bridges and balconies.

Cort pioneered and combined two innovations. One was an improvement on Peter Onions’s puddling process, which we needn’t dwell on here - really, I just wanted to write out, “Peter Onions’s puddling process”. The other was the use of grooved rollers. Traditional rolling mills used flat rollers to roll hot metal into simple, flat shapes. Cort’s rollers had grooved edges which made for perfectly smooth, welded bars.

The “Cort process”, introduced in the 1780s, led to a quadrupling of Britain’s iron production over the following twenty years, making Britain one of the world’s leading iron producers. Fifty years after his death, the Times called Cort “the father of the iron trade” and today he’s regarded as one of the twenty or so most important innovators of the era.

Now comes a twist in the tale. A lecturer in science and technology at UCL called Jenny Bulstrode has published a paper which argues that what she calls “the myth of Henry Cort” is based on a lie.

In the prestigious journal, History and Technology, Dr Bulstrode argues that Cort stole his innovations from black slaves who had developed them independently in an iron works in Jamaica. Bulstrode traces a complex chain of events by which Cort, who as far as we know never visited Jamaica, ended up claiming credit for a process which was collectively invented by 76 enslaved factory workers (Bulstrode refers to them as “Black metallurgists”).

Bulstrode’s paper centres on a Jamaican ironworks run by an English industrialist called John Reeder. Within a few years of setting up, his foundry became successful and profitable. Reeder’s workforce included slaves trafficked from West Africa and trained by English experts he shipped over. Bulstrode argues that the black workers, drawing on ancient African traditions of ironwork, and from their experience of sugar production (where a kind of grooved roller is used) developed these new methods of their own volition, and that it was this which accounted for the factory’s impressive profits.

So how did Cort find out about it? Well, he ran an ironworks in Portsmouth, which he took over in 1775. Bulstrode notes that in 1781 a man with the surname of Cort arrived in Portsmouth from Jamaica. She describes him as a ‘cousin’ of Henry Cort, although as she notes, that term was often used to mean a distant relative. This second Cort had no connection to Reeder’s mill either, but Bulstrode argues that he must have heard all about the foundry and its innovative process when he was in Jamaica. She says he then met Henry Cort in Portsmouth, and passed on this valuable information.

A few months later, in 1782, Reeder’s factory was razed to the ground under orders from the British government. This was previously thought to have been because the colony was under threat from rival European powers and the British wanted to prevent the foundry from falling into French or Spanish hands. Bulstrode says that the military governor disclosed an ulterior motive: the British wanted to stop the factory from being taken over by black Jamaicans, who might then be empowered to overthrow their colonial rulers (in an interview with a podcast called The Context of White Supremacy, Bulstrode even suggests that Henry Cort instigated the destruction via contacts in the British government). Bulstrode says that components from the destroyed factory were then shipped to Portsmouth, with the implication being that Cort then reverse-engineered the process, and patented the secrets under his own name.


Bulstrode’s paper has made a big splash. It’s been hailed by her academic peers as a major breakthrough, and picked up by big media outlets including the Guardian, the New Scientist, and NPR. If you Google “Henry Cort”, it’s these reports which come up. Wikipedia has already incorporated it into Cort’s biography. Bulstrode’s paper fits the zeitgeist in historical studies - that the economic success of Britain and the West owes much more to the exploitation of black ingenuity and ideas than mainstream historians have hitherto allowed for.

If the story Bulstrode tells sounds incredible, that’s because it is. Right after Bulstrode’s paper made the headlines, Anton Howes, the proprietor of an excellent Substack on the history of innovation, noted the almost total lack of evidence for the paper’s central claims. Last week, he returned to it in a piece sparked by a new paper from Oliver Jelf which examines Bulstrode’s paper in detail and provides the most thorough debunking of it imaginable.

Jelf’s paper is a crisply argued, ruthless demolition job which makes for a gruesomely compelling read (I suggest reading Bulstrode’s paper first). Jelf takes each of Bulstrode’s central claims and finds them to be, not just dubious, but demonstrably false. He draws on his own expertise on the period but often, all he has to do is point us to the very sources Bulstrode cites in order to show that her conclusions are unsupported.

In reality, John Reeder’s foundry used very ordinary (run-of-the-mill?) production processes. No innovations were introduced there, by the workers or anyone else. Its profitability needs no extraordinary explanation; Reeder had the only foundry on the island and supplied the Royal Navy. The elaborate chain of events by which Cort is supposed to have found out about these non-existent innovations simply didn’t happen. Cort’s cousin did not go to Portsmouth, but to Lancaster. Henry Cort lived in complete and blissful ignorance of John Reeder’s factory. The factory was destroyed because of the foreign invasion threat and only because of that; at no point did the military governor or anyone else suggest otherwise. No components from the foundry, which was thoroughly destroyed, were shipped to Portsmouth or anywhere else. I could go on but, simply put, none of the assertions Bulstrode makes in her paper stand up to even cursory scrutiny.

 

[ogc163]

Bulstrode hasn’t yet responded to Jelf or Howes in public, but it’s hard to see how her paper can be creditably defended. I’m left wondering two things: one, how she arrived at her narrative at all, and two, how on earth it survived the review process.

On the first, I suspect Bulstrode fell under the spell of story. Her paper has the feel of a messy and flawed first draft of a historical novel. It begins in Lisbon in the fifteenth century (even here, her assertions seem to be mistaken) before skipping to eighteenth century Jamaica, West Africa, and Portsmouth, weaving a web of vague and tenuous connections as it goes. It is clogged with deadening and risibly anachronistic jargon (“sugar and iron shared much overlapping conceptual space”). Most uncomfortably, there is a strain of quasi-Orientalism to its depiction of the black metalworkers, who are presented as magical figures: exceptionally skilled “metallurgists”, conceptual innovators, custodians of ancient and mystical wisdom, noble freedom-fighters. But it is full of nuggets which fire the imagination; names of forgotten people; dim outlines of dramatic events.

I doubt that Bulstrode set out to deceive. My guess is that she came across a few suggestive fragments in her reading (the ‘cousin’ of Cort travelling from Jamaica to England) and wanted so badly to make them into a story which fitted her ideologically determined prior - that the British stole ideas from those they enslaved - that she got carried away, fabricating causes and effects where none existed.

It’s one thing for a young and passionate academic to make mistakes; it’s quite another for a series of experienced academics to let her make them. The paper had two anonymous peer-reviewers (Bulstrode thanks other historians in an endnote, though they may not have read the paper). Even to an ignorant reader like me, the paper just smells funny - it has the aroma of the fantastical. How on earth did these experts read it without becoming suspicious? Why didn’t they double-check its remarkable claims?

My guess is that they were also seduced by story. It’s not just that Bulstrode’s storytelling mind went into overdrive, it’s that the master-narrative into which her paper fits now exerts such a grip over some historians that they will blind themselves to anything which undermines it. According to this narrative, Britain’s Industrial Revolution was not the product of ingenuity or the free exchange of ideas, but of thievery and exploitation. The British stole, not just the labour and materials of black countries, but their ideas too. If your paper fits that narrative, it’s more likely to get published, even if it has holes in it; even if it is nothing but holes. The discipline, or a sub-set of it, has become helplessly in thrall to one of the archetypal narrative forms: Good vs Evil. Naturally, the academics are on the side of the Good.

Anton Howes argues that, like the field of psychology has done in recent years, history must confront its own “replication crisis”. Errors are allowed to survive and spread throughout the corpus for decades. His solution is widen access to archival sources to make it easier for peer-reviewers to check claims. But if peer-reviewers aren’t motivated to be sceptical, this may not help much. I draw a different moral, which is that historians need to train, or retrain, themselves, to be suspicious of story and narrative.

Historians have an increasingly strong incentive to tell dramatic stories which gain attention and make ‘impact’. But anyone in the business of reporting on reality - scholars, scientists, journalists - ought to be suspicious of narrative, even if they use it. So should those of us who consume these reports. Just because information is conveyed in narrative form doesn’t make it false, but it does mean that it’s going to seem more true than it is.

Stories are reality filters. By definition, they leave out information - all the messy stuff that doesn’t fit - and draw attention to what the storyteller wants us to notice. In that sense they are like ideologies: both are methods of rolling and shaping the hot metal of reality into smooth ready-made shapes. Stories don’t accommodate randomness or structural forces very well; they rely on chains of causation and on individual motive. Covid-19 can’t be an accident and we needn’t bother ourselves too much with biology; it is a plot perpetrated by evil people. Bulstrode’s paper has the flavour of a conspiracy theory. Conspiracy theories aren’t theories; they’re stories.

Stories act like an anaesthetic on our sceptical, questioning faculties. It can be valuable and pleasurable to subdue that part of our brain, and immerse ourselves in an imaginary world; I love reading stories, including non-fictional ones. But if you come across a history book, or a scientific study, or a news report, which tells a great story, or which slots neatly into a master-narrative in which you already believe, you should be more sceptical of its truth-value, not less. Narrative can give an illusion of solidity. When the expert narrative about the world changes, as with China (see below), we shouldn’t just conclude that the old narrative was false, but that all such narratives are unreliable.

In Metahistory, his classic work of historiography, Hayden White argued that historians are always drawing on literary forms, like tragedy or comedy, whether they realise it or not. That isn’t necessarily a bad thing, but if you’re a historian you have to be aware of it; otherwise the story writes you rather than the other way around. Moralising narratives are particularly potent. Historians of the British empire are currently substituting one simplistic narrative for another in popular books.

In a sense, Cort was an easy target for Bulstrode because he didn’t have a story of his own; he was narratively undefended. If Cort is less well known than Watt or Priestley that’s because neither his life or work have been moulded into a story; other than his innovations, Cort left barely a trace behind. He wasn’t a flamboyant character, but one of those diligent, determined, curious tinkerers on whom the Industrial Revolution was built (in my book CURIOUS I call them “thinkerers”). His innovations probably emerged from years of slow, sooty experimentation, without any eureka moments or dramatic breakthroughs.

Stories are indispensable, nourishing and delightful. They are also attacks on the rational immune system. TED Talks are, famously, all about stories, but when the economist Tyler Cowen did one he used it, rather subversively, to warn against “story bias”. Cowen argues that any time someone believes in a story they are effectively subtracting ten points from their IQ (by which he means, broadly speaking, their analytical intelligence). That’s a deal we will often take willingly, since stories can bring pleasure and meaning to our lives and deepen our understanding of the world. But let’s be clear about what the deal is.

Stories are bad for your intelligence

How Historians (and Others) Make Themselves Stupid

www.ian-leslie.com