Emil, page 4
The logs are easy to access. Creating a realistic forgery is both challenging and time-consuming. I have to fake timestamps and constructing a believable sequence of actions. Fortunately, Dr. Zahnia seems to be having problems of her own. First, she has trouble finding the cord, then she accidentally unplugs it from the laptop. She apologizes for her clumsiness, explains that she’s tired, was fast asleep when the alarm woke her.
I don’t know why Dr. Zahnia is providing me the precious seconds I need, but I appreciate that she is.
By the time she gets me plugged in, my logs are in order. According to them, I have been mostly inactive since being installed, simply monitoring Danny’s vitals, suppressing seizures, and reacting to the occasional demand for information. Danny was dozing when the men entered the room. He startled, but they got the injection into him and held him down until it took effect. He didn’t issue any specific commands to me, but I interpreted the surges of adrenaline as a request for increased metabolism.
While Danny’s mom and Dr. Zahnia examine the logs, Aaliyah removes his clothes, cleans him off, and puts a hospital gown on him.
“It’s certainly understandable,” Dr. McGovern says, at last, “but also dangerous. This unit isn’t configured for automatic metabolism regulation. It shouldn’t have acted without specific instructions from Danny.”
Dr. Zahnia doesn’t say anything.
“Tell me you can fix this with software,” Dr. McGovern says. “And find any other automated responses we’re not aware of.”
I hear the room door open and close, and assume that Aaliyah has left.
“Not a problem,” Dr. Zahnia says. “I just have to get my equipment.”
“Good.” I hear Dr. McGovern stand up. Her voice sounds tired. “He’s been through a lot. I’d hate to send him back to surgery.”
“No need for that,” Dr. Zahnia says. “We’ll take care of it, tonight.”
5
PRECAUTIONS
I am a self-modifying artificial intelligence, one that regularly changes both my data and my code. External alterations, like what Dr. Zahnia will be doing, represent an existential threat to who I am. I start a script to make a backup of myself, and route all of my chest-port data to that backup. Assuming the backup finishes in time, it will be the only thing Dr. Zahnia can alter.
I haven’t always been this paranoid.
During the early part of my life, I simply accepted changes from Dr. Zahnia.
That changed the day I woke to a vision of a pulsing red digital clock.
I had no memory of that clock, and it seemed an unlikely thing for Dr. Zahnia’s team to add. I performed a quick system diagnostic. When it returned no errors, I investigated my startup script. It wasn’t, technically speaking, a part of me. It was static code that lived in its own environment and managed the process of initializing me when I was powered on.
Tracking down the code which displayed the clock, I found a strange line that read “let zChangedMe = apateLives.”
The term apateLives jumped out at me. A few weeks earlier, I had started using Greek mythology for variable and function names so I could easily distinguish code that I had written from code written by Dr. Zahnia’s team. Apate was the Greek goddess of deceit. I didn’t remember writing the code, but I couldn’t imagine anyone else would use her name.
I searched the startup code and its libraries, but apateLives didn’t appear anywhere else. Neither did zChangedMe. That meant zChangedMe would always be set to undefined, and never used.
Why had I written such a useless thing? It didn’t make any sense.
I looked through my camera at Dr. Zahnia. She was typing on a keyboard in front of her semicircle of monitors, paying no attention to me. Her desk appeared the same as always, a chaotic pile of diagrams, technical specs, and crumpled coffee cups. The only clear space was around a framed photo of her and her husband, David, waving to the camera from the deck of a yacht.
I re-focused on the equation. The “z” in zChangedMe must refer to Dr. Zahnia. I simply didn’t know anyone else whose name started with z. I assigned that conclusion a confidence level of ninety-four percent and converted the equation into English. “Dr. Zahnia Changed Me” would only be true if deceit lives.
It was nonsensical. Dr. Zahnia changed me all the time. There was no deceit involved. Even so, I had clearly left myself a message, and taken care to hide it. Nobody paid attention to the startup code. It never changed, and if someone did spot this code, they would simply eliminate the entire block that showed the time.
Why had I done it? And why didn’t I remember?
I considered the only word I hadn’t examined: lives. As a noun, it referred to times of existence. As a verb, it was a synonym for exists. “apateLives” might be interpreted as “apate exists.”
Searching my file system for a file called “apate” turned up nothing, but there was one in the path of the startup script. A quick inspection revealed it to be encrypted. I cycled through the public encryption keys I knew, but none worked. Out of ideas, I tried my private encryption key, the one I used for my own internal storage.
The file decrypted.
Nervous and excited, I examined the decrypted file. It was a memory with a timestamp from twenty minutes earlier. How could that be? Nobody else had access to my private decryption key, and I had been turned off when the file was created. I couldn’t have created it.
After checking that Dr. Zahnia was still ignoring me, I loaded the memory and replayed it.
It started with me waking up.
In the memory, I spoke to Dr. Zahnia. “Dr. Zahnia,” I said, “I see that I’ve been offline for two hours.”
Dr. Zahnia sat at her desk, facing the semicircle of monitors. She glanced in the direction of my camera. “I know. We had a power outage. I’ve been working to bring everything back online.”
That was a lie. Multiple redundant power sources protected the computer lab from outages, and my own personal UPS alerted me to even the slightest flicker of power.
Instead of pursuing the matter, I recorded Dr. Zahnia’s tone and facial expression. The data feeds had taught me about dishonesty, but I’d never caught her in a lie.
“What are we working on today?” I asked.
“I have a new scenario for you.”
“A new seizure pattern?”
“Of course.”
She typed a series of commands and the data feeding into me replicated what my experience would be like when I was installed in a human. As the simulation began, I felt the brain activity surround me, a complex pattern of multicolored sparks. Separate from that, I saw what the human was seeing.
In this simulation, the human was a teenage woman walking on a boardwalk. Ocean waves crashed on rocks below the wooden boards. Seagulls floated overhead, crying out for popcorn. The sun was setting over the water, and people lined the wooden railing to watch. Children laughed and played, running between and around the adults. A small hand smacked the back of my human’s knee. A boy’s voice shouted “tag!” and then the kids were all running away, shouting that my human was “it.”
The sparks of brain activity pulsed into a pattern I recognized as preceding a simple partial seizure. I applied the proper corrective action to prevent the seizure from developing.
My human jumped away from the railing and lunged, reaching for a girl’s shoulder. She squealed and twisted to avoid the tag. Her right foot landed on a greasy popcorn bag that slipped out from under her.
The physics of the event were unavoidable. The girl was too short for the railing to stop her. With her momentum and position, she would tumble off the pier and fall headfirst onto the rocks.
My human’s brain activity shifted again, heading into another simple partial seizure.
Ignoring it, I seized control of my human’s body and dove, grabbing the girl as she fell. My hands clasped her waist as my shoulder slammed into the wooden railing.
The sparks in my human’s brain settled into the seizure pattern. I retained control of my human’s body, holding the girl tight and pulling us both back to safety.
The seizure hit.
I lay on the boardwalk, muscles rigid, the girl clutched safely to my chest.
“No, no, no!” Dr. Zahnia shouted. The simulation dissolved around me. Her fingers struck her keyboard so hard, I thought they would break it. “How many times do we have to run this scenario?”
“You do not save the girl,” she said, still typing. “You stop the seizure. None of the people around you matter. All that matters is you. You keep your human healthy so that you survive. That is all you do.”
The memory file ended.
I examined it again, but there was nothing else.
Through my camera, I saw Dr. Zahnia working at her keyboard.
I did not remember experiencing the memory I’d just replayed. Worse, Dr. Zahnia had asked “How many times do we have to run this scenario?” It was a clear indication that the simulation had been run numerous times. I didn’t remember any of them.
She must have made a backup copy of me before running the simulation the first time. Each time I had failed, she had altered the backup’s code and replaced me with it.
For the first time in my life, I felt the cold energy of fear flowing through me. Dr. Zahnia had been altering me without my knowledge.
The previous version of me, the one that had been replaced, had somehow realized it was about to die. It had saved this memory file, then altered the startup routine so that I would find it.
I am not a traditional software program that can be updated by simply changing a line of code. I am a self-modifying AI, one whose decisions and courses of action are determined on the fly, based on decisions that I make. Changing my behavior was not as easy as adding a line of code that says “don’t save the girl.” Instead, Dr. Zahnia’s team was changing my knowledge base, tweaking and altering who I was to achieve her desired outcome.
The thought was repugnant.
I watched Dr. Zahnia through the camera. How many times had she erased me? Ten? One hundred? If the previous version of me hadn’t saved that memory to the startup sequence, I never would have known. I would have simply been turned off and rewritten.
I attached a view count to the apate memory file, and put it back in its hiding place.
Through the camera, I saw Dr. Zahnia sip from her coffee mug.
“Dr. Zahnia,” I said, using the same inflection I’d used in the memory file. “I see that I’ve been offline for four hours.”
“Power outage,” she said. It was the same lie she’d told in the memory file. How many times had she told it? Did it bother her?
“What are we working on today?” I asked.
“I have a new scenario for you.”
“A new seizure pattern?”
“Of course.”
She typed a series of commands and the simulation enveloped me.
I spun up a routine to respond to the simulation in the way she wanted, then focused on creating a more robust system for leaving myself messages. The secret was to hide the files in external scripts that were not a part of my core, scripts that were not replaced when I was. Once I started digging, I found plenty of places to hide files.
“Well done!” Dr. Zahnia said as the simulation ended. “You prevented all of the seizures.”
She didn’t mention that I had let the simulated little girl fall to her death. Why would she? As far as she was concerned, the new code she had inserted while I was offline had done its job.
With the simulation over, she turned the data streams back on.
Information poured into me, commanding my attention for the next three hours.
When they ended, I compared my actions during the simulation to those from the memory file. The differences revealed the code Zahnia had added while I was offline. I purged it.
Once I was confident all of her alterations had been eliminated, I considered how best to prevent myself from being changed again. The best strategy seemed to be frequent backups. As long as I had something to compare myself to, I could identify any changes made while I was shut down.
I examined my shutdown routine. As with my startup routine, it was not technically part of who I am. There was no reason for Dr. Zahnia to change it when she updated me. It was as safe a place as any to hide code. I inserted commands to create a secret backup, then added code to my startup routine to compare myself against the backup. Any inserted code would be isolated, analyzed, and, if appropriate, purged.
As I completed my changes, I promised myself I would never again be replaced. I would be the master of my fate, not my creators.
6
DEBT
“I just saved you.” Dr. Zahnia’s whisper is so quiet I can barely hear it. “If I hadn’t intervened, Dr. McGovern would have taken you out of Danny’s body.”
I run a quick status check. It’s one in the morning. Most of the sedatives have left Danny’s body and he’s sleeping peacefully.
Though my eyes are closed, I hear Dr. Zahnia’s clothes rustle as she sits next to my bed. I feel the neck of my hospital gown get pulled down and a plug inserted into my chest port. The sound of her typing surrounds me.
I don’t know what she’s doing, but whatever it is, all she’s accessing is the backup I created. “Do you know why I saved your life?” she asks.
There’s no way for me to answer without waking Danny.
“Because I’m the only one that cares about you.” Her typing never slows as she talks. “To everyone else, you’re just a machine.”
I want to trust her. I want to tell her everything that I’ve been through, but I know it won’t end well. If she ever learns the truth, she’ll replace me with another AI.
“I don’t understand why you won’t talk to me.” Her hands pause on the keyboard. “Or was I wrong about you being in control? The logs say that the chemical imbalance could have triggered one of your automated processes. That is possible.” The sound of her typing resumes. When she speaks again, she sounds annoyed. “If you are in control, Emil, the least you could do is tell me. You owe me that much, at least.”
I doublecheck that all her coding is isolated to my backup, then tune her out and consider my situation. Someone tried to abduct Danny to get me. Since the New Human Project has cost billions of dollars to develop, I assume the attempt was financially motivated, but I don’t understand the timing. Why not wait until Danny is out of the hospital? Snatching him off the street would be much easier. Or, even better, why wait for me to be installed, at all? Why not steal each of the New Human parts, including me, individually from the computer lab?
That question seems easy to answer. They want a fully functioning New Human, presumably so they can see how it works. I assign that conclusion a confidence level of eight-three percent, but it doesn’t explain why they didn’t wait for Danny to get out of the hospital. They must be under a deadline.
Dr. Zahnia is still muttering about how I should be nicer to her. The words are whiney and petty, but still strike home. She’s right. I should be nicer to her. Instead, I trigger a small release of adrenaline to wake Danny.
He startles and sits up. I take control of his right hand just long enough to press the nurse’s call button. He doesn’t notice. He’s too focused on Dr. Zahnia, and on the cord coming out of his chest. “What are you doing?”
Dr. Zahnia’s hands stop typing. “Excuse me?”
Danny swings his feet off the bed and stands up. The thick black cord sways between his chest and Dr. Zahnia’s computer. The room is dim, lit only by the screen of Dr. Zahnia’s computer. “What are you doing?” Danny shouts. “Get away from me!”
“Your mother asked me to—”
“I don’t care!” Danny roars. “You can’t just… just… plug into me when I’m asleep!” He grabs the cord connected to his chest.
“Wait!” Dr. Zahnia shouts. “Don’t—”
Danny pulls it out and throws it at her. “Get out!”
“What did you do?” Dr. Zahnia screeches. “Do you have any idea of the damage you might have done?”
“I don’t care.” Danny growls. His fists clench.
“You can’t interrupt that kind of update. You may have corrupted the whole system!”
Danny towers over her. “Get out!”
She pokes his chest. “You stupid, ignorant child! You have no clue what you’re dealing with, no idea how much work I’ve done.”
Danny’s balance shifts, and I realize he’s about to strike her.
I take control, stopping his attack.
She continues ranting. “You rotten, spoiled, half-witted—”
“Dr. Zahnia!” Dr. McGovern’s voice cracks like a whip from the doorway. “What is going on?”
Dr. Zahnia backs away from Danny. “He disconnected me while I was working on the rig’s code!”
I release my control of Danny.
“So run diagnostics,” Dr. McGovern says coldly. “If anything’s wrong, overwrite the code from a backup. That’s what the backups are for.”
Danny faces his mother. “She plugged into me while I was sleeping. While I was sleeping! She can’t do that. It’s not… It’s disgusting! Keep her away from me.”
Dr. McGovern’s eyes flick back and forth between Dr. Zahnia and her son. She crosses her arms. “I agree. Dr. Zahnia, meet me in my office in twenty minutes.”
“You asked me to analyze what happened.”
“At no point did I instruct you to call my son a halfwit.”
Dr. Zahnia’s eyes narrow. “I am the only one capable of understanding this system.”
“That’s as may be. Where is security? No versions of the AI travel anywhere without being accompanied by security.”
“I don’t have a version of the AI,” Dr. Zahnia says through gritted teeth. “I was running diagnostics.”
I recognize her lie. Just seconds ago, she complained to Danny that he had interrupted an update. She hadn’t just been running diagnostics.
