Why Do Extraordinary Claims Require Extraordinary Evidence?
Carl Sagan made this anti-BS test famous. But people widely misunderstand what it means, and how to apply it.
Nearly half a century ago, astronomer Carl Sagan popularized a concise test for separating science from pseudoscience. “Extraordinary claims require extraordinary evidence,” he declared repeatedly, in interviews, books, and his landmark TV show, Cosmos.
At the time, Sagan was primarily taking aim at tabloid stories about space aliens visiting Earth. Since then, his dictum has embedded itself in the popular culture as an all-purpose rule for shooting down wild, unsubstantiated claims: everything from QAnon conspiracy theories to online hype about alleged breakthroughs like cold fusion and warp drives. The “extraordinary evidence” standard has also provoked a great deal of ongoing confusion and dissent—even among academic scientists and philosophers.
One common objection is that there should be just a single standard of evidence of all claims. After all, the argument goes, who is to say that a particular claim is “extraordinary”? And what makes some evidence ordinary whereas other evidence qualifies as extraordinary?
I recently spoke with Carol Cleland, a prominent philosopher who studies the definition of life, who raises another objection to Sagan’s quote. She notes that researchers often invoke the “extraordinary evidence” standard to dismiss evidence for the existence of extraterrestrial life. Some of these researchers go further, though, and suggest that the notion of extraterrestrial life is inherently “extraordinary,” and that the default position therefore should be that extraterrestrial life does not exist. “Isn’t that claim also extraordinary?” Cleland asks.
Detangling this conceptual mess is important. Modern life is full of nonsense, and “extraordinary claims require extraordinary evidence” is a powerful tool for filtering out the BS. But, as with all tools, it must be used correctly to be effective.
Extraordinary? Says Who?
The confusion around the Sagan test centers on the word “extraordinary,” and on the ways it is routinely (and often wildly) misunderstood.
The intrepid folks at Quote Investigator trace the roots of the “extraordinary evidence” quote back to a 1708 essay by a British rector named Benjamin Bayly. In his essay, Bayly advised skepticism toward anyone claiming to receive messages directly from God. “Sure, these Matters being very extraordinary, will require a very extraordinary Proof,” he wrote. Four decades later, philosopher David Hume articulated a similar test for evaluating claims of miracles. In this view, an extraordinary claim is one that is literally beyond the ordinary—that is, a claim that doesn’t accord with the laws of nature as they are known.
Another common interpretation of the Sagan test is that it’s a variant on Ockham’s Razor, the principle that, when faced with a number of possible explanations, the simplest one is likely to be the best one. In this view, an extraordinary claim is one that demands a more complicated explanation, and so goes against the principle of simplicity.
You may notice, though, that both of these interpretations don’t really get to the vexing questions at the heart of the matter: Who gets to decide what is an extraordinary claim, and what kind of extraordinary evidence is required to support it?
Fortunately, there is a specific, concrete, and quite straightforward way to define both “extraordinary claims” and “extraordinary evidence.”
If you want to sound super-smart, you can say that the answer is rooted in Bayes’ theorem, first articulated by the 18th century statistician and minister Thomas Bayes. If you want to be a bit more direct, you can call it conditional probability. More directly still, you can simply call it prior knowledge: The likelihood that a claim is true depends on how much knowledge we already have that would support that claim.
The Power of Prior Knowledge
Assessing the probability of a claim based on prior knowledge is a type of statistical analysis with a well-formulated set of rules. A lot of the time, though, you can access the core principles of conditional probability using nothing more than common sense. Everyday experience has already wired a lot of Bayesian analysis into your brain. Here’s an example of what I mean:
If I say, “I met my brother yesterday, and here’s a photo of our meeting,” you would probably accept that statement with no question. But now I ask you to step back and ask yourself, why? Why does my claim seem so ordinary and believable?
The answer is that you already have a huge body of prior knowledge to draw on. You know that humans exist. You know that humans often have brothers. Even if you don’t know that I have two brothers, you could easily confirm that by calling my parents or looking up my birth certificate. My birth certificate can confirm that two other people who share my last name were born in the same hospital, to the same parents, within a few years of each other. You know that adult humans often visit their siblings, especially if their siblings live nearby. I am an adult human. My brother lives nearby. And on and on.
These details sound so mundane, so idiotic, that you might not even think of them as prior knowledge—but they are. Each detail supports my simple statement that I met with my brother yesterday. My claim (I met my brother yesterday and took a photo of him) seems ordinary because you already have an extraordinary body of evidence in support of its plausibility.
If I now say, “I visited with an intelligent space alien yesterday, here’s a photo,” that’s a superficially similar statement. I visited someone, I took a photo, I am showing you the photo. Two similar claims should require similar evidence, right? Sure, if both statements existed in a vacuum, without any other knowledge to draw on. But that is not even remotely the case.
In the case of the space alien, you lack prior knowledge at every level. You have no evidence of alien life. You have no evidence that some alien life is intelligent. You have no evidence that intelligent space aliens can travel between the stars. You have no evidence that they travel to Earth. You have no evidence that they have met with humans. We have no evidence that they visited humans where I live. And on and on.
Making Sense of Extraordinary Claims
Here, then, is a simple way to break down the Sagan test:
1. Every claim requires an abundance of well-documented evidence to be considered plausible.
2. “Extraordinary evidence” is another way of describing that abundance of well-documented prior knowledge, or an equivalent abundance of well-documented new information.
3. An “extraordinary claim” is a claim made with little or no prior knowledge supporting it.
If we already possess abundant prior knowledge, the claim qualifies as ordinary (“I met with my brother”). That supporting evidence may seem mundane as hell, but that does not make its quantity and depth any less extraordinary.
If we do not already possess abundant prior knowledge, the claim qualifies as extraordinary (“I met with a space alien”).
The probability that an extraordinary claim is true remains low unless the person making the claim can provide an abundance of novel evidence so iron-clad that prior knowledge is not necessary. For instance, if I could present the space alien and submit it to tests showing that it does not share DNA or basic biological structures with other creatures on Earth, that would be quite convincing. It would then rise to the standard of extraordinary evidence.
Once you reframe the Sagan test this way, everything falls into place. All claims must meet the exact same standard of extraordinary evidence (or you can call it an abundance of well-documented prior knowledge, if you prefer—same thing). A claim is ordinary if you already have that kind of evidence in hand. A claim is extraordinary if you do not. It’s that simple.
Now you can see why Carol Cleland and many other scientists bristle when they hear people say that it is a waste of time to search for alien life, since it probably doesn’t exist. I agree with Cleland: The claim that alien life does not exist is just as unsupported as the claim that it does. On this topic, the only well-supported claim is that we do not yet know whether or not alien life exists.
More important, now you can see where the real double standard occurs. Lots of conspiracy-minded folks argue that, for instance, UFO sightings should be treated just as credibly as any other kind of sighting. They insist that a claim lacking extraordinary evidence (I saw a UFO) should be granted the same status as a claim that has such evidence (I saw my brother).
In other words, the UFO believers set a special, extremely low standard of evidence for things that they want to be true…simply because they want them to be true. This attitude is common across pretty much all conspiracy beliefs.
A final note. It’s instructive to read the longer version of Carl Sagan’s quote, as it appears in his book Broca’s Brain: “I believe that the extraordinary should certainly be pursued. But extraordinary claims require extraordinary evidence.”
The investigation of extraordinary claims is a crucial part of how science progresses. Researchers seek out phenomena that go against our current understanding of the world—that is, seemingly contrary to prior knowledge. If those contrary phenomena are decisively observed, they falsify the old theory and point the way to a new, more fully descriptive one.
Want to convince me that space aliens are real? I would love to believe you. It would completely transform our understanding of life and of the place of our species in the universe. But in the absence of prior knowledge of aliens, you better bring some very convincing new evidence to the table. Extraordinary evidence, you might say.