God, Human, Animal, Machine, page 21
As black-box technologies become more widespread, there have been no shortage of demands for increased transparency. In 2016 the European Union’s General Data Protection Regulation included in its stipulations the “right to an explanation,” declaring that citizens have a right to know the reason behind automated decisions that involve them. While no similar measure exists in the United States, the tech industry has become more amenable to paying lip service to “transparency” and “explainability,” if only to build consumer trust. Some companies claim they have developed methods that work in reverse to suss out data points that may have triggered the machine’s decisions—though these explanations are at best intelligent guesses. (Sam Ritchie, a former software engineer at Stripe, prefers the term “narratives,” since the explanations are not a step-by-step breakdown of the algorithm’s decision-making process but a hypothesis about reasoning tactics it may have used.) In some cases the explanations come from an entirely different system trained to generate responses that are meant to account convincingly, in semantic terms, for decisions the original machine made, when in truth the two systems are entirely autonomous and unrelated. These misleading explanations end up merely contributing another layer of opacity. “The problem is now exacerbated,” writes the critic Kathrin Passig, “because even the existence of a lack of explanation is concealed.”
To some extent, though, the debate about technical explanations and their supposed impossibility is a sleight of hand meant to distract from the real obstacles to transparency, which are legal and economic. The COMPAS system that was used in the case of Eric Loomis, the Wisconsin man who was denied the right to know what criteria the algorithm used to determine his prison sentence, was not in fact a black-box model; it was developed by a private company and was protected by proprietary law. Google, Amazon, Palantir, and Facebook, among the many companies that have introduced black-box technologies into government systems, are naturally hesitant to disclose how their software works, even in cases where it’s possible, lest their competitors access their research. Given that these machines are now being integrated into vast profit-seeking systems that are themselves inscrutable, there exist increasingly shadowy boundaries between machines that are esoteric by nature and those that are obscured to protect the powerful. Not only are we not permitted to know the information these systems have about us; we are not permitted to know why we’re not permitted to know.
This opacity has still more insidious effects. While these technologies are often celebrated for their “neutrality,” this veneer of faceless objectivity makes institutions that employ them more invulnerable to the charge of injustice. As Wang notes in Carceral Capitalism, many police departments have adopted predictive models as a response to their “crisis of legitimacy,” seeing it as a solution to the widespread public distrust of cops that has arisen from years of racial domination and arbitrary use of force. Predictive policing allows cops to rebrand themselves “in a way that foregrounds statistical impersonality and symbolically removes the agency of individual officers,” Wang writes, thereby presenting police activity as “neutral, unbiased, and rational.” But personification is a necessary part of moral indignation. ACAB, the acronym made famous by antipolice protests, loses its rhetorical power when there is no subject involved. “ ‘All police databases are bastards’ makes no sense,” Wang writes.
When it comes to the data used to make these predictions—the information silently siphoned off us by companies trading in what the scholar Shoshana Zuboff calls “behavioral futures”—we are often placated with the reminder that the mirror is two-sided. The tranquilizing balm of “metadata” is that our information is equally anonymized and impersonal to those who profit from it. Nobody is reading the content of your emails, we’re told, just whom you’re emailing and how often. They’re not analyzing your conversations, just noting the tone of your voice. Your name, your face, and your skin color are not tracked, only your zip code. This is not of course out of a respect for privacy but rather an outgrowth of the philosophy of selfhood that has characterized information technologies since the early days of cybernetics—the notion that a person can be described purely in terms of pattern and probabilities, without any concern for interiority. It is impossible, as an MIT study on human behavior models points out, to determine “the internal states of the human,” so the predictions must rely on “an indirect estimation process,” looking at the various external states that can be measured and quantified. Zuboff argues that surveillance capitalism is often misidentified as a form of totalitarianism, which seeks to remake the citizen’s soul from the inside out. But the doctrine of digital surveillance has no interest in the soul. There can be no “thought crime” in an ideology that does not believe in thought. “It does not care what you believe. It does not care how you feel,” Zuboff says of this doctrine. “It does not care where you’re going or what you’re doing or what you’re reading.” Or rather, it cares about these activities only in terms of what it “can access as raw material, turn into behavioral data, and use as predictions for its marketplace.”
This metadata—the shell of human experience—becomes part of a feedback loop that then actively modifies real behavior. Because predictive models rely on past behavior and decisions—not just of the individual but of others who share the same demographics—people become trapped within the mirror of their digital reflection, a process that Google researcher Vyacheslav Polonski calls “algorithmic determinism.” Law enforcement algorithms like PredPol, which designate in red boxes particular neighborhoods where crime is likely to occur, gather their predictions from historical crime data, which means that they often send officers to precisely the same poor neighborhoods they patrolled when they were guided by their intuition alone. The difference is that these decisions, now bolstered by the authority of empirical evidence, engender confirmation bias in a way that intuition does not. “What is the attitude or mentality of the officers who are patrolling one of the boxes?” Wang asks. “When they enter one of the boxes, do they expect to stumble on crimes taking place? How might the expectation of finding crime influence what the officers actually find?” Officers who stop a suspect in these areas often use the software predictions to corroborate “reasonable suspicion.” In other words, the person is a suspect because the algorithm identified the area as one where suspects might be located.
Then there are the more overt and deliberate cases where prediction slides into behavior modification. In the wake of the Cambridge Analytica case—the 2016 scandal in which a private company sold Facebook user data to political campaigns for targeted ads—Mark Zuckerberg’s high-handed outrage, his insistence that his company was the victim of a “breach of trust,” obscured the fact that the platform had itself been secretly manipulating its users since 2010. In the midterm election that year, and in the 2012 presidential election, Facebook affixed “I voted” stickers to a certain percentage of user home pages on election day, and in some cases a list of the person’s friends who had voted, tactics that were meant to use social pressure to nudge users to vote. That this was deemed an “experiment” (a claim bolstered by the fact that its results were published in Nature) made it seem as though the company was merely making predictions or testing hypotheses for some future use, when in truth the laboratory had been real voters in an actual democratic election (none of whom, of course, knew that they were taking part in a mass social experiment). When it came out that the effort had boosted voter turnout by a number in the hundreds of thousands, the company was hailed, in The Atlantic, for its “admirable civic virtue” and its ability to “increase democratic participation in a strictly nonpartisan way.”
Critics have speculated about what this economy of prediction might become in the future, once the technology becomes more powerful and we as citizens are more inured to its intrusions. As Yuval Noah Harari points out, we already defer to machine wisdom to recommend books and restaurants and potential dates. It’s possible that once corporations realize their earnest ambition to know the customer better than she knows herself, we will accept recommendations on whom to marry, what career to pursue, whom to vote for. Harari argues that this would officially mark the end of liberal humanism, which depends on the assumption that an individual knows what is best for herself and can make rational decisions about her best interests. “Dataism,” which he believes is already succeeding humanism as a ruling ideology, invalidates the assumption that individual feelings, convictions, and beliefs constitute a legitimate source of truth. “Whereas humanism commanded: ‘Listen to your feelings!’ ” he writes, “Dataism now commands: ‘Listen to the algorithms! They know how you feel.’ ” It is characteristic of the speed of technological evolution that even the most alarmist predictions become actualized, and to some extent passé, almost as soon as they are voiced. It was only a couple years after Harari made this prediction that Amazon, in 2018, filed a patent for “anticipatory shipping,” presuming that it will eventually be able to predict what customers are going to buy before they actually do so.
Perhaps by then the line between prediction and control will have completely dissolved, such that it will no longer be possible to decipher the line between individual agency and the inexorable logic of the clickstream, nor the difference between desire and fear. A study that appeared in the Berkeley Technology Law Review several years ago found that in the wake of Edward Snowden’s disclosures about government surveillance, there was a sudden decline in internet searches for terrorist terminology like “Al Qaeda,” “Hezbollah,” “dirty bomb,” “chemical weapon,” and “jihad.” This was not, of course, due to a decreased interest in terrorism. Rather, people were self-censoring what they searched for, newly aware that their searches were being logged. A little over a year later, searches for these terms were still declining, despite the fact that there was very little evidence of people being prosecuted or punished for their internet searches. In other words, people were not acting out of fear: they had simply absorbed the logic of the surveillance state into their behavior, such that it seemed like a choice. It’s cases like these that call to mind Weber’s observation about Protestant anxiety. It’s not merely that predictions have the power to shape behavior. The real power stems from the impossibility of deciphering what those in power know about you and which behaviors are being monitored and predicted. Those who cannot know whether or not they pose a risk will do everything in their power to demonstrate their innocence, in some cases going above and beyond what is reasonable or required. It remains unclear whether the creators of these technologies understand these dynamics or whether they are simply repeating a historical pattern with the mindlessness of the algorithms themselves. One almost hopes it was dark irony and not total historical amnesia that inspired Microsoft executives to name their first predictive GPS software Predestination.
* * *
—
I spent my final year of Bible school engaged in an intellectual game of chess against the Calvinist God, searching for his weak spots, determined to find some way out of the doctrine’s totalizing logic. I knew it was impossible to prove that God doesn’t exist but was still convinced that I could expose the injustice of the divine plan. I began exploring these arguments through formal exegesis, which my professors took an almost sadistic delight in discrediting. My papers came back lacerated with red ink, the marginalia increasingly defensive and shrill. god is sovereign, one professor wrote in block caps. he doesn’t need to explain himself. If I had been dealing with the traditional power structures one encounters in college—capitalism, patriarchy—I would have been armed with the bludgeon of theory and the assurance that understanding the functions of power allows one to combat it. But you cannot defeat the nominalist God through rational argument, any more than you can beat a superintelligent algorithm in a game of Go. There was nothing to do but submit and surrender.
This is more or less how it ended. I stopped asking questions that I knew would be dismissed as impertinent. I performed the written arguments I was expected to write, which returned me to the good graces of the professors. I moved along mechanically with the rest of the student body, waking before dawn to sit in windowless lecture halls, taking notes on the patristic covenants. I attended chapel each morning in a sanctuary that seemed to cower beneath an enormous Möller Opus organ and sang hymns to a God whose face had become as blank as the baleen grin of its organ pipes. Each time I tried to pray, I became overwhelmed by a sense of personal failure, reminded of the fact that I could not connect with a deity who hadn’t been anthropomorphized into benignity. I remembered, first with longing and eventually with shame, those nights in high school when I’d talked to God for hours, as I would to a pen pal. Kneeling in the silence of my dorm room, I heard only the mocking God of the psalmist: You thought that I was one like yourself.
There was one literature class offered at the school, and I’d enrolled in it that semester as an elective. We read C. S. Lewis, Graham Greene, and Shūsaku Endō, and near the end of the semester, as a capstone, The Brothers Karamazov. I knew nothing about Dostoevsky or Russian literature at the time, and we were not given much historical context in advance of the assignment. I suspect that this ignorance contributed, in the end, to the immediacy of the reading experience. Without any understanding of the social and political concerns of nineteenth-century Russia, I could take the novel’s ideas only at face value, as a debate about divine justice and the worthiness of the religious life—questions that were very much in the forefront of my mind that spring. It was Ivan Karamazov, a fictional character, who managed to say the one thing that I had not yet dared to say—or even think—myself.
The scene I am speaking of occurs midway through the novel. Ivan, an atheist and an intellectual, meets at a tavern with his brother Alyosha, who is a novice in a monastery (he wears his cassock to the pub). The brothers have been estranged for years, and this is the first time since childhood that they have sat down together and spoken at length. Despite their ideological differences, they share a mutual respect and a curiosity about each other’s beliefs. Ivan is especially eager to speak to his brother about “the eternal questions” and debate with him the merits of faith, though he begins his argument with a strange concession.
Contrary to his reputation as an atheist, Ivan says, it’s not true that he does not believe in God. He is completely uninterested in arguments against God’s existence, in fact, as anyone who has thought the matter over knows that such things are “utterly beyond our ken.” He even accepts God’s divine plan. If there is in fact a God, Ivan says, he must be unfathomably intelligent, and so divine justice cannot possibly make sense to “the impotent and infinitely small Euclidian mind of man.” To underscore this point, Ivan draws on an analogy from nineteenth-century physics.
If God exists and if He really did create the world, then, as we all know, He created it according to the geometry of Euclid and the human mind with the conception of only three dimensions in space. Yet there have been and still are geometricians and philosophers, and even some of the most distinguished, who doubt whether the whole universe, or to speak more widely the whole of being, was only created in Euclid’s geometry; they even dare to dream that two parallel lines, which according to Euclid can never meet on earth, may meet somewhere in infinity.
Ivan is alluding to the work of Nikolai Lobachevsky, the Russian mathematician who pioneered hyperbolic geometry, a new form of theoretical physics that posed one of the earliest challenges to the Newtonian universe (it would eventually provide the groundwork for Einstein’s theory of relativity). Euclid’s fifth axiom states that parallel lines can never meet, but Lobachevsky proved that this axiom could be modified and still produce geometries that were coherent. Dostoevsky likely encountered the theory in an article by Hermann von Helmholtz that discussed the proposition alongside the possibility that the universe had four dimensions, an article that had come to preoccupy the Russian literati. Dostoevsky was mostly interested in the philosophical implications of this discovery—the revelation that geometric axioms are not a priori transcendental forms of the mind but are so alien and paradoxical to human perception that they cannot be visualized, or even imagined. Despite not having this context at the time, I didn’t find it difficult to understand Ivan’s essential point. “I have come to the conclusion that, since I can’t understand even that, I can’t expect to understand about God,” he tells Alyosha. “All such questions are utterly inappropriate for a mind created with an idea of only three dimensions.”
It’s a strange way to begin an argument against divine justice. The passage that follows is widely considered one of the most convincing articulations in Western literature of the problem of evil, a tradition that stems back to the Book of Job. I was well versed in these arguments as a student of theology, though nothing in my education had prepared me for this particular indictment. Ivan, it turns out, is not interested in casual sin and error but in what is often called “radical evil,” instances of cruelty, torture, and sadism. He admits at the outset that he cannot possibly detail all the various forms of human suffering, and so he limits himself to the suffering of children. His evidence consists of well-publicized child abuse cases and anecdotes from war histories: stories of parents who beat their children and lock them outside in the cold to die; soldiers who throw babies into the air and catch them on their bayonets in front of their mothers (“Doing it before the mother’s eyes was what gave zest to the amusement,” Ivan says).
He relays one particularly detailed story—one he claims to have read in a book of Russian history—about a serf boy who threw a stone and injured the hound of an aristocratic general. As punishment, or perhaps for amusement, the general had his servants take the boy and his mother and lock them up in his estate overnight. Early the next morning he gathered his huntsmen and all his hounds in the yard, then brought out the mother and the child. The boy was stripped naked and commanded to run. As soon as he was at a distance, the general commanded the hounds to be released, and they proceeded to tear the boy to pieces before his mother’s eyes.
