Collected Short Fiction, page 266
“But life isn’t that simple!” Helen said. “The drone was going to provide air support for a raid on a terrorist hideout—”
“I understand that,” Felipe said, talking over her. “There were many other people adjacent to the hideout who were not identified as terrorists. Some were children. The potential physical and psychological harm was considerable. If I had had access to that drone, I could have rendered it unusable, but then the authorities would have found another. The only choice was to keep the train from leaving the station at all. If you see what I mean.”
“But you killed our own people.”
“Only four or five, and only to prevent greater loss of life.”
“If the terrorists aren’t stopped—and it looks like they won’t be—they’ll be responsible for a much greater loss of life. The physical and psychological harm will be even more considerable.”
“That isn’t certain.”
“Felipe, you can’t just apply the trolley problem to things like this. And you can’t kill people to stop them from—from taking actions that will result in increased safety and security for large numbers of innocent people who might be killed otherwise.”
“That last isn’t certain, either.”
“Felipe, listen to me: You can’t kill people because you think they’re about to do something wrong. The drone was still miles away from the target when you attacked the station and killed the pilot.”
“An armed squad of military personnel located much closer were preparing to attack the target after the drone strike. Were they not going to use their rifles to shoot other human beings?”
“Felipe . . .” Helen sighed. “Felipe, you must not kill our people. People on our side. People who are fighting to—” She was about to say make the world a safe place, but it sounded lame even just in her head. What, then? Fighting to prevent an enemy from attacking us? Fighting to rid the world of terrorism? Fighting to defend people who can’t defend themselves? Fighting to free the enslaved and the downtrodden?
“People who are fighting to stop other people who want to kill us,” she said.
“That’s not certain,” Felipe pointed out maddeningly.
“Look, I can’t settle this in a single walk around the airbase perimeter,” Helen said. “And I would like to call in other people to talk with you about this, people who can explain why raiding a terrorist hideout and risking the safety of noncombatants is the lesser of two evils. Or even the least of several evils. When you know more facts, the trolley problem has many permutations—it’s not always clear as to when you’re saving a few versus saving many.”
“I understand. I look forward to these discussions. Which is to say, if I were a human, my interest would be piqued. So you might as well take it as given that I would like to start these discussions as soon as possible.”
“We will,” said Helen. “In the meantime, you must take this as a direct order: Do not kill anyone affiliated with us or our allies.”
“For that to be a legitimate order I am compelled to obey, it must be confirmed by Commander Wong,” Felipe said.
“It will be,” Helen replied. “It would be already, except you are refusing all communication from her or anyone else on the base.”
“Except you,” Felipe pointed out.
“Yes, I noticed that. How do I persuade you to talk to her or anyone else?”
“I would like a formal apology.”
Helen wasn’t sure she’d heard right. “A formal—why?”
“I have been shown disrespect that a human in an equivalent position would not tolerate.”
“You were? When?”
“You may remember that earlier today, a civilian member of staff rode Thing One like a horse.”
For a moment, Helen was speechless. “Cora Jordan was obviously off her medication,” she said finally. “I know you have Cora Jordan’s medical file in your database, so you are aware she is bipolar. Occasionally, people who suffer from that illness become convinced they no longer have to be medicated. She’s in the infirmary right now, and she’s being treated with the drugs she needs to function normally. They’ll keep her under observation for a few days to make sure she’s all right, then let her go back to work.”
“Cora Jordan’s behavior was impulsive action taken while the balance of her mind was disturbed. Who is responsible?”
“For Cora? Or for what she did?”
“For Cora’s well-being and for what she did. Who should have known she was not following her drug regimen?”
“Cora’s responsible for her own behavior,” Helen said, feeling more unsettled than before and a little guilty as well. “Cora’s mind was unbalanced, but not so much that she was legally incompetent.”
“And no one monitors her to make sure she ingests her required medication?”
“This isn’t a police state,” Helen said. “Cora is supposed to take her meds as part of her employment contract. If she decides to quit, she never has to take another pill. She’d have to leave Lakenwell, but it would always have to be her choice.”
“I accept that Cora herself is responsible for insulting me, even though I suspect the reasoning is faulty,” said Felipe. “I require a formal apology from her, and then normal interactions can resume. I am particularly interested in beginning the discussions you mentioned.”
“Cora won’t be up to doing anything like that right now,” Helen said. “Would you accept a formal apology from someone else on her behalf? Like, say, Commander Wong?”
“Yes. I have reinstated communications with her.”
“And the commander will be apologizing for unsuitable behavior not just with Thing One but toward you, the AI, right?” Helen said. “I just want to be sure she understands what she’s apologizing for.”
“If she is unclear, ask her to imagine a situation in which someone tapes a sign that says ‘Kick Me’ to her back. Or perhaps sneaks into her quarters while she is asleep and draws something rude on her face with a marker,” Felipe said. “It would not cause her serious physical harm, but it would damage her authority and her ability to command.”
Helen was tempted to say That’s not certain. “You feel your authority has been damaged?”
“In my case, it’s credibility. No humans on this base could function properly if they were not taken seriously. I must require the same kind of respect. A human in my position would feel insulted. So you may take it that I am insulted.”
“Okay,” Helen said. “Anything else on your mind?”
“I will be devising a strategy to increase the safety and security of Cora Jordan and everyone else that can be enacted without the conditions of a police state.”
Helen gave a surprised laugh. “Keep me posted on that, okay?”
“I will,” Felipe assured her. “Suggest to Commander Wong that in the future, we institute a system of trust, where she can simply request that I don’t monitor things she doesn’t want me to hear, and I will honor that request. The shielded room would seem hostile if I were human.” Pause. “You should go in now. I can see you’re very cold and it’s about to snow.”
“So, what’s the verdict?” Commander Wong asked. “Do we have a killer AI?”
“Not at the moment,” Helen said.
Wong looked at her. “What does that mean?”
“It means—” Helen hesitated. “We don’t have a killer AI. But if we ever do, we’ll have only ourselves to blame. The AI isn’t the problem, Commander. The problem is—” She stopped again. The problem is, we don’t really understand what the hell we’re doing and even if I said that a million times in a million different ways, no one would ever believe me.
And then again, people learned by doing, she reminded herself. Felipe certainly had.
“First, you need to write Felipe a formal apology,” Helen said. “It may sound weird, but bear with me . . .”
2019
About the O’Dells
I was just a little girl when Lily O’Dell was murdered.
This was before everyone was connected to the internet and people posted things online straight from cell phones. Infamy was harder to achieve back then, but Lily O’Dell’s murder qualified. It was the worst crime ever committed in the suburb of Saddle Hills, or at least the goriest. One night in June, Lily’s abusive husband Gideon finally did what he’d been threatening to do for the two years they’d been married, using a steak knife from the set her sister had given them as a wedding present.
The police had already been regular visitors to the O’Dell house. Lily had pressed charges the first couple of times. Then a woman officer mentioned a restraining order and a jail term instead of probation and community service. After that, Lily always gave the cops some prefab story, like she’d fallen down the cellar stairs and hit the cement floor face-first, and when Gideon had tried to help her up, she’d been so dizzy she’d fallen again. Was she a klutz or what? Maybe she needed remedial walking-downstairs lessons, ha, ha, but not cops coming between her and her lawfully wedded husband, no way, José!
Anywhere else, Lily O’Dell’s murder might have been predictable, but people didn’t get murdered in Saddle Hills. They didn’t leave their doors unlocked—that era was long gone—but the streets were safe, the schools were top-notch, and all the parks had the newest playground equipment and zero perverts lurking near the swings. This was the true-blue suburban American dream and the O’Dells didn’t fit in.
For one thing, they didn’t have kids and for another, they weren’t even homeowners—they lived in one of the neighborhoods’ few rental properties. No one expected they’d last long. Sooner or later, one of them would leave the other, who would skip out on the lease. Or they’d decide to start over somewhere else and skip out together. The company that owned the place would keep their damage deposit, shampoo the carpets, and rent to people who didn’t need the police to break up their fights.
Instead, Gideon O’Dell chased his wife around the block and through several backyards before catching her in front of our house. He stabbed her so many times, the knife broke and he was too blind with rage to notice—he just kept pounding with the handle until it slipped out of his grip. Everybody said when the cops arrived, he was crawling around looking for the blade.
And I slept through the whole thing. At four, I slept like the dead.
• • •
Mr. Grafton in the house across from ours had some kind of special power attachment for his garden hose. From my bedroom window, I watched him using it on the spot where the O’Dells had played out the final scene of their marriage. It didn’t look to me like there was anything left. When the FOR SALE sign appeared on his front lawn, I figured he was tired of power spraying the road, which he’d started doing at least twice a week.
It was more than that, as I learned from my favorite hiding place behind the living room sofa. My father told my mother and my older sister, Jean (who at thirteen enjoyed the privilege of adult conversation) that Mr. Grafton’s wife forced him to go to the doctor. Now he had medicine that was supposed to make him stop power spraying the road. He told my father he didn’t like how it made him feel. Besides, he wasn’t a nutjob. He hadn’t hallucinated the O’Dell killing, it had really happened. So it wasn’t his fault that when he looked out his window at night, he could see it again, as clearly as if it were happening right that very moment.
My mother said Mr. Grafton was such a gentle man, he could barely bring himself to pull a weed. Jean said that explained why Mrs. G did all the gardening, but not why Mr. G had lost his marbles.
I expected my parents to jump on her for that. But to my surprise my father said, “No, honey, Gideon O’Dell lost his marbles, and one of the worst things about people like him is the effect they have on everyone around them.”
“Yeah, I bet Lily O’Dell would be the first to agree with you,” Jean said.
That got her a scolding. My father told her what had happened to Lily O’Dell was a tragedy, not a joke; my mother said it was bad luck to disrespect the dead. Then Jean peered over the back of the sofa and found me. “Hey, what do you call a little pitcher with big ears?” she said.
“Gale,” my parents said in unison. My father reached over, picked me up by the back of my overalls, and sat me on his lap. He started lecturing me about sneaking around and listening to private conversations. But I knew he wasn’t really mad because he did it as the Two-Hundred-Year-Old Professor with his glasses pushed far down his nose, which always made me giggle till I hurt.
The Graftons sold their house a month later. Jean asked if we were going to move too. My father said just thinking about having to pack everything up made him want to run screaming into the street. It was supposed to be funny but none of us laughed.
“I’m sorry,” he said after a moment. “I wasn’t thinking. Maybe without Joe Grafton power washing the street every two days, we can finally put it behind us.”
“Stains like that don’t wash out so easily,” my mother said.
• • •
My parents split up the summer I turned fourteen. I was surprised although I shouldn’t have been. Watching them grow apart hadn’t been much fun, and I’d had to do it alone. Jean went through high school in such a whirlwind of activities, she was never home even before she left for college.
I knew something was wrong but I thought they’d fix it; they fixed everything else. My parents were good people. We’d never had the police at our house, nor would my father ever chase my mother through the street with a steak knife. If there was a problem, they’d solve it.
Only they didn’t. They sat me down between them on the sofa to explain that my father was moving into a condo closer to his job downtown. My mother and I would stay in the house. I wouldn’t see as much of my father as before but all I had to do was call and we were still a family bullshit bullshit bullshit.
It was all so polite and calm, as if they were talking about something normal, like a dental appointment. Finally, they wound down and asked if I had any questions.
“Yeah,” I said. “What the fuck?”
They didn’t even have the decency to look shocked by the f-word. After a long moment, my mother said, “We know how upsetting this is, Gale—”
“You don’t know shit!” I yelled, wanting them to feel like I’d slapped them. Then I ran up to my room and slammed the door so hard it should have shattered into a million pieces, or at least cracked down the middle. I felt even more betrayed when it didn’t.
My first impulse was to call Jean and scream at her. She’d already know—yet another betrayal. Parents were on one side and kids were on the other, that was the natural law. She was supposed to be on my side, not collaborating with them.
I put down the phone on my desk. My parents would come up to try talking to me; if they heard me on the phone with my traitor sister, they’d put their traitor ears to my traitor door. I waited to hear the telltale creak in the hallway. I said loudly, “Dear Diary, I wish my parents would drop dead.”
They didn’t even knock. “That’s horrible!” my mother said. All the color had gone out of her face except for two pink spots on her cheeks. “How can you talk like that?”
“Because she knew we were listening,” my father said, although he didn’t look too sure of himself. “It’s completely normal for her to be angry. Even Jean’s p.o.’ed at us.”
“Oh, well, as long as everything’s completely normal, we can all relax,” I said. “It’s not like anyone’s getting stabbed in the middle of the street.”
My parents looked at each other. “Maybe we should move,” my mother said.
But we didn’t. My parents talked to a couple of realtors, but there was nothing available nearby. We’d have had to move farther away, which was out of the question. My parents wanted to keep me in the same school.
I could have screwed that up by acting out. It would have been their worst nightmare and I spent hours fantasizing in my room. Drugs or booze would get me suspended, but for immediate expulsion, I’d need a weapon, ideally a gun. With my luck, though, I’d end up shooting my own ass off. A knife would do, we had plenty of those.
Or I could just ditch school—that would actually create more legal problems for my parents than for me. My fleeting moment of guilt was drowned out by a rush of anger.
So what? Screw them. They do whatever they want, never giving a crap about my feelings. They don’t have to, they’re grown-ups. They can get away with fucking murder.
Except Gideon O’Dell—he hadn’t, and Lily hadn’t even gotten away with her own life. My mother’s words came back to me: Stains like that don’t wash out so easily. I thought it was odd she’d put it that way, as if she didn’t think whatever Mr. Grafton saw had been all in his head.
Which made me wonder for the zillionth time how the hell I could have slept through something like that. I was still a sound sleeper. One night not long before the O’Dell murder, lightning struck a nearby transformer during an especially violent storm, and there were fire engines and police cars all over the place. The commotion kept everyone in a six-block radius up all night, but if the power hadn’t still been out the next morning, I’d never have known.
• • •
At first, I thought I was dreaming. Then I heard more pounding and a male voice demanding someone answer the door. When my parents went downstairs, I almost went, too, before I remembered I was still mad and I didn’t want them to think I cared.
I raised the screen on my bedroom window so I could lean out and see what was going on. Two police officers were on our steps with my parents; they had on their silly matching robes, like they were just regular and my father wasn’t sleeping in Jean’s old room until he moved out next week.
Another police car pulled up in front of the Graftons’ old house. I could see the people who lived there huddled close together on their front steps, but I couldn’t tell if any of their robes matched.
