Weaponized Lies: How to Think Critically in the Post-Truth Era, page 19
Suspect Guilty
YES
NO
Evidence Match
YES
0.95
0.45
1.4
NO
0.05
8.55
8.6
1
9
10
We know from our forensics team that the probability of a match for the hair sample is .95. Multiplying that by one, we get the entry for the upper left, and subtracting that from one we get the entry for the lower left. If there is a .95 chance that the sample matches the victimized horse, that implies that there is a .05 chance that the sample matches a different animal (which would absolve the suspect) so the upper right-hand cell is the product of .05 and the marginal total of 9 = .45. Now when we perform our calculations, we see that
P(Guilty | Evidence ) = .68
P(Evidence | Guilty) = .95
P(Innocent | Evidence) = .32
P(Evidence | Innocent) = .05
The new evidence shows us that it is about twice as likely that the suspect is guilty as that he is innocent, given the evidence. Many attorneys and judges do not know how to organize the evidence like this, but you can see how helpful it is. The problem of mistakenly thinking that P(Guilty | Evidence) = P(Evidence | Guilt) is so widespread it has been dubbed the prosecutor’s fallacy.
If you prefer, the application of Bayes’s rule can be done mathematically, rather than using the fourfold table, and this is shown in the appendix.
FOUR CASE STUDIES
Science doesn’t present us with certainty, only probabilities. We don’t know for 100 percent sure that the sun will come up tomorrow, or that the magnet we pick up will attract steel, or that nothing travels faster than the speed of light. We think these things very likely, but science yields only the best Bayesian conclusions we can have, given what we know so far.
Bayesian reasoning asks us to consider probabilities in light of what we know about the state of the world. Crucial to this is engaging in the kind of critical thinking described in this field guide. Critical thinking is something that can be taught, and practiced, and honed as a skill. Rigorous study of particular cases is a standard approach because it allows us to practice what we’ve learned in new contexts—what learning theorists call far transfer. Far transfer is the most effective way we know to make knowledge stick.
There is an infinite variety of ways that faulty reasoning and misinformation can sneak up on us. Our brains weren’t built to excel at this. It’s always been a part of science to take a step back and engage in careful, systematic reasoning. Case studies are presented as stories, based on true incidents or composites of true incidents, and of course, we are a story-loving species. We remember the stories and the interesting way they loop back to the fundamental concepts. Think of the following as problem sets we can all explore together.
Shadow the Wonder Dog Has Cancer (or Does He?)
We got our dog Shadow, a Pomeranian-Sheltie mix, from a rescue shelter when he was two years old. He got his name, we learned, because he would follow us from room to room around the house during the day, never far away. As often happens with pets, our rhythms synchronized—we would fall asleep and wake up at the same time, get hungry around the same time, feel like getting exercise at the same time. He traveled with us often on business trips to other cities, becoming acclimated to planes, trains, and automobiles.
When Shadow was thirteen, he began having trouble urinating, and one morning we found blood in his urine. Our vet conducted an ultrasound examination and found a growth on his bladder. The only way to tell whether it was cancerous was to perform two surgical procedures that the oncologist was urging: a cystoscopy, which would run a miniature camera through his urethra into the bladder, and a biopsy to sample the mass and study it under the microscope. The general practitioner cautioned against this because of the risks of general anesthesia in a dog Shadow’s age. If it did turn out to be cancerous, the oncologist would want to perform surgery and start chemotherapy. Without any further tests, the doctors were still pretty certain that this was bladder cancer, known as transitional cell carcinoma (TCC). On average, dogs live only six months following this diagnosis.
As my wife and I looked into Shadow’s eyes, we felt utterly helpless. We didn’t know if he was in pain, and if so, how much more he was facing, either from the treatment or from the disease. His care was entirely in our hands. This made the decision particularly emotional, but that didn’t mean we threw rationality out the window. You can think critically even when the decision is emotional. Even when it’s your dog.
This is a typical medical scenario for people or pets: two doctors, two different opinions, many questions. What are the risks of surgery? What are the risks of the biopsy? How long is Shadow likely to live if we give him the operation and how long is he likely to live if we don’t?
In a biopsy, a small needle is used to collect a sample of tissue that is then sent to a pathologist, who reports on the likelihood that it is cancerous or not. (Pathology, like most science we’ve seen, does not deal in certainties, just likelihoods and the probability that the sample contains cancer, which is then applied to the probability that the unsampled parts of the organ might also contain cancer; if you’re looking for certainty, pathology is not the place to look.) Patients and pet owners almost never ask about the risk of biopsy. For humans, these statistics are well known, but they are less well tracked in veterinary medicine. Our vet estimated that there was a 5 percent chance of life-threatening infection, and a 10 percent chance that some cancerous material (if indeed the mass was cancerous) would be “shed” into the abdomen on the needle’s way out, seeding further cancer growth. An additional risk was that biopsies leave behind scar tissue that makes it more difficult to operate later if that’s what you decide to do. The anesthesia needed for the procedure could kill Shadow. In short, the diagnostic procedure could make him worse.
Our vet presented us with six options:
Biopsy through the abdominal wall in the hope of obtaining a more definitive diagnosis.
Diagnostic catheterization (using a catheter to traumatize a portion of the mass, allowing cells to exfoliate and then be examined).
Biopsy using the same cystoscopic camera they wanted to use anyway to better image the mass (through the urethra).
Major surgery right now to view the mass directly, and remove it if possible. The problem with this is that most bladder cancers return within twelve months because the surgeons are unable to remove every cancerous cell, and the ones left behind typically keep on growing at a rapid rate.
Do nothing.
Put Shadow to sleep right now, in recognition of the fact that it most probably is bladder cancer, and he doesn’t have long to live anyway.
We asked about what the treatment options were if it was found to be cancer, and what they might be if it was not cancer. Too often, patients focus on the immediate, upcoming procedure without regard for what the next steps might be.
If the mass was cancerous, the big worry was that the tumor could grow and eventually block one of the tubes that brings urine into the bladder from the kidneys, or that allows urine to leave the bladder and end up on a lawn or fire hydrant of choice. If that blockage occurs, Shadow could experience great pain and die within a day. Along the way to this, there could be temporary blockages as a result of swelling. Because of the position of the bladder within the body, and the angle of ultrasound, it was difficult to tell how close the mass was to these tubes (the ureter and urethra).
So what about the six options presented above—how to decide which (if any) to choose? We ruled out two of them: putting Shadow to sleep and doing nothing. Recall that the oncologist was pushing for surgery because that is their gold standard, their protocol for such cases. We asked for some statistics and she said she’d have to do some research and get back to us. Later, she said that there was a 20 percent chance that the surgery would end badly, killing Shadow right away. So we ruled out the major surgery because we weren’t even sure yet if the mass was cancerous.
We asked for life-expectancy statistics on the various remaining scenarios. Unfortunately, most such statistics are not kept by the veterinary community, and in any case, those that are kept skew toward short life expectancy because many pet owners choose euthanasia. That is, many owners opt to put their pets down before the disease progresses because of concerns about either the animal’s quality of life or the owners’ quality of life: Dogs with TCC often experience incontinence (we had already noticed that Shadow was leaving us little surprises around the house). We didn’t have a definitive diagnosis yet, but based on the sparse statistics that existed, it looked as though Shadow would live three months with or without treatment. Three months if we do nothing, three months if we give him chemo, three months if we give him surgery. How could that be? Ten years ago, we found out, vets would recommend euthanasia on first diagnosis of TCC. And at the first sign of chronic incontinence, owners would put their dogs down. So owners were typically ending their dogs’ lives before the cancer did, and this made the statistics unreliable.
We did some research on our own, using “transitional cell carcinoma” and “dog or canine” as the search terms. We found out that there was a 30 percent chance Shadow could improve simply by taking a nonsteroidal anti-inflammatory called Piroxicam. Piroxicam has its own side effects, including stomach upset, vomiting, loss of appetite, and kidney and liver trouble. We asked the vet about it and she agreed that it made sense to start him on Piroxicam no matter what else we were doing.
From the Purdue University website—Purdue has one of the leading veterinary medical centers—we were able to obtain the following survival statistics:
Median survival with major surgery = 109 days
Median survival with chemotherapy = 130 days
Median survival with Piroxicam = 195 days
The range of survival times in all of these studies, however, varied tremendously from dog to dog. Some dogs died after only a few days, while others lived more than two years.
We decided that the most rational choice was to start Shadow on the Piroxicam because its side effects were relatively minor, compared to the others, and to get the cystoscopy in order to give the doctor a better look at the mass and the associated biopsy to give us more to go on. Shadow would have to be lightly anesthetized, but it was only for a short time and the doctors were confident that he would emerge fine.
Two weeks later, the cystoscopy showed that the mass was in fact very close to the ureters and the urethral openings—so close, in fact, that surgery wouldn’t help if the mass was cancerous because too much of the tumor would be left behind. The pathologist wasn’t able to tell if the tissue was cancerous or not because the procedure ended up not getting a large enough sample. So after all that, we still didn’t have a diagnosis. Yet the statistics above suggested that if Shadow was among the 30 percent of dogs for whom Piroxicam worked, that would yield the best life expectancy. We wouldn’t have to subject him to the discomforts of surgery or chemo, and we could just enjoy our time together at home.
There are many instances, with both pets and humans, that a treatment doesn’t statistically improve your life expectancy. Taking a statin if you are not in a high-risk group or surgically removing the prostate for cancer if you do not have fast-moving prostate cancer are both treatments with negligible impact on life expectancy. It sounds counterintuitive, but it’s true: Not all treatments actually help. It’s clear that Shadow would be better off without the surgery (so that we could avoid the 20 percent chance it would kill him) and the chemo wouldn’t buy him any time, statistically.
Shadow responded to the Piroxicam very well and within three days he was back to himself—energetic, in a good mood, happy. By one week he had no more difficulty urinating. We saw occasional minor amounts of blood in the urine, but we were told this was normal after biopsy. Then, 161 days after the initial suspicion (which was never confirmed) of TCC, his kidneys started to fail. We checked him into a specialty oncology clinic. The doctors weren’t sure whether the organ failure might be related to TCC, or why it was occurring now. They prescribed medications to address common kidney conditions and ran dozens of tests without getting any closer to understanding what was happening. Shadow grew increasingly uncomfortable and stopped eating. We put him on an IV drip painkiller and two days later, when we took him off for just a few minutes to see how he was doing, he was clearly in pain. We talked to his current and former doctors, carefully describing the situation, its progression, and his condition. All agreed it was time to let him go. We had Shadow’s company—and he had ours—for a month longer than the average chemotherapy patient, and during that month he was able to avoid hospitals, catheters, IV lines, and scalpels.
We went to the oncology hospital—the staff knew us well because we had been visiting Shadow there every day in between his tests and treatments—and arranged for him to be put to sleep. He was in pain, and we felt that we had perhaps waited one or two days too long. It was awful to see that large personality suddenly drift away and disappear. We found comfort knowing that we had considered every stage of his care and that he had as good a life as we were able to give him for as long as possible. Perhaps the most difficult emotion that people experience after a disease ends a life is regret over the choices made. We were able to say good-bye to Shadow with no regrets over our decisions. We let our critical thinking, our use of Bayesian reasoning, guide us.
Were Neil Armstrong and Buzz Aldrin Thespians?
Moon-landing deniers point to a number of inconsistencies and unanswered questions. “There should have been more than a two-second delay in communications between the Earth and the moon, because of its distance.” “The quality of the photographs is implausibly high.” “There are no stars in the sky in any of the photos.” “How could the photos of the American flag show ripples in it, as though waving in the air, if the moon has no atmosphere?” The capper is a report by an aerospace worker, Bill Kaysing, who wrote that the probability of a successful landing on the moon was .0017 percent (note the precision of this estimate!). Many more such claims exist. Part of what keeps counterknowledge going is the sheer number of unanswered questions that keep popping up, like a game of Whac-A-Mole. If you want to convince people of something that’s not true, it’s apparently very effective to simply snow them with one question after another, and hope that they will be sufficiently impressed—and overwhelmed—that they won’t bother to look for explanations. But even 1,000 unanswered questions don’t necessarily mean that something didn’t happen, as any investigator knows. The websites dedicated to the moon landing denial don’t cite the evidence for it, nor do they publish rebuttals to their claims.
In the case of the moon landing, each of these (and the other claims) is easily refuted. There was a two-second delay in Earth-moon communications that can be easily heard on the original tapes, but some documentary films and news reports edited out the delay in the interest of presenting a more compelling broadcast. The quality of the photographs is high because the astronauts used a high-resolution Hasselblad camera with 70mm high-resolution film. There are no stars in the lunar sky because most of the images we saw were taken during lunar daytime (otherwise, we wouldn’t have been able to see the astronauts). The flag doesn’t show ripples: Aware that there was no atmosphere, NASA prepared the flag with a t-bar to support its top edge and the “ripples” are simply folds in the fabric. With no wind to blow the flag, its creases stay in place. This claim is based on still photos in which there appears to be a rippling effect, but moving film images show that the flag is not blowing, it’s static.
But what about the report of an aerospace worker that a moon landing was highly improbable? First, the “aerospace worker” was not trained in engineering or science; he was a writer with a BA in English who happened to work for Rocketdyne. The source of his estimate appears to be from a Rocketdyne report from the 1950s, back when space technology was still in its infancy. Although there are still unanswered questions (e.g., why are some of the original telemetry recordings missing?), the weight of evidence overwhelmingly points to the moon landing being real. It’s not certainty, it’s just very, very likely. If you’re going to use spuriously obtained probability estimates to claim that past events didn’t happen, you’d have to similarly conclude that human beings don’t really exist: It’s been claimed that the chances of life forming on Earth is one in many billions. Like many examples of counterknowledge, this uses the language of science—in this case probability—in a way that utterly debases that fine language.
Statistics Onstage (and in a Box)
David Blaine is a celebrity magician and illusionist. He also claims to have completed great feats of physical endurance (at least one was recognized by the Guinness Book of World Records). The question for a critical thinker is: Did he actually demonstrate physical endurance or was he using a clever illusion? Certainly, as a skilled magician, it would be easy for him to fake the endurance work.
