Society of the Mind, page 20
"How long is an 'inordinate period of time'?"
"Well that makes sense for just this one keyboard, but presumably you're processing things nonstop. You're constantly coordinating all those digital computers and updating your model of the world. I watched Griffith at breakfast 'channel surf' through thousands of different scenes from your cameras. You should be seeing things happening all the time."
"What kind of things do you see but not notice?"
"Would you have any memory at all of the object you saw but didn't notice?"
At lunch, Laura took her regular place at the table in the computer center's conference room — at the right hand of Gray, the master.
Gray surprised the team, which had gathered for lunch, by calling on Laura first. He surprised Laura most of all. She was famished and had just taken a huge bite of her sandwich.
"The computer…" Laura began with her mouth full, but had to pause in order to wash her food down with a Coke. "Excuse me." She cleared her throat and dabbed at her lips with a napkin.
"The computer uses a 'generate-and-test' model of observation. Stimuli from its cameras and microphones and whatever are constantly processed to test its hypotheses about the world. If it detects a change — if it sees someone, for instance, where it didn't think anyone was before — it updates its model. But it still hasn't 'noticed' anything. To determine whether the change comes to the computer's attention, it uses what we call a 'sentry system.' The boards that updated its world model fire the new observation out randomly through their connections. If another board cares about the change — like a board in the security system that sees someone where they're not supposed to be — then the computer notices what the camera saw. When the computer's attention is drawn in that manner, it gives the event a date and timestamp through a process called content-sensitive settling and then records it as a memory."
"So does that mean you think the computer is conscious?" Hoblenz asked, seemingly impatient with her report.
"Well, it depends on what you mean by 'conscious.' The computer's behavior is not just lifelike, it's like life of a very high order. Everyone understands a sunflower to be 'alive,' for example. It even exhibits what appears to be sophisticated behavior. As the sun moves across the sky, the sunflower turns to follow it. But it's just a plant. When the sun goes behind a cloud, the sunflower can't anticipate where the sun will be when it emerges. An animal, however, maintains a 'real representation' of environmental features even when they aren't in direct sensory contact. When a lion sees a zebra, the lion doesn't forget where the zebra is when it looks away momentarily. Your computer works the same way."
Laura assumed they probably already knew what she'd just reported, but her "teammates" refrained from deriding her this time. That fact buoyed her — made her feel a more qualified member of the team. "So," an emboldened Laura said in the silence, "how goes the effort to load that antiviral program?"
Filatov shrugged. "We've cut way down on system maintenance and are leasing third-party computers to off-load operations. We hope to free up enough capacity to try the phase-two again sometime tonight."
"Any new ideas about what's wrong with the computer?" Laura asked in between bites of her sandwich.
"We have competing theories in play," Gray answered. "Mr. Hoblenz believes we're being hit with penetration attempts by governmental or corporate intruders." Hoblenz nodded.
"Dr. Bickham, however," Gray continued, looking at Margaret, "has raised the relatively frightening prospect of a malignant, mutant virus that evolved naturally from the computer's ecosystem."
Laura was poised to take another bite of her sandwich, but she lowered it from her mouth and said, "Whoa, whoa. What did you just say?"
"Do you know what genetic algorithms are?" Gray asked.
Laura hesitated. "If I say no, am I gonna get a long lecture?"
Hoblenz laughed in a low and raspy voice. The only other person at the table to smile was Gray. "Well, it's important for your work that you know." Laura sighed and nodded. She sank back into her chair and crossed her arms over her chest.
"We've achieved three breakthroughs in natural computation that have allowed us to build an intelligent computer. The first was the architecture — a true neural network. The second was successfully programming into the computer the rules of fuzzy logic and chaos theory, which allow the computer to predict what was previously thought to be unpredictable. But the third was our use of genetic algorithms, which was by far the most important advance. Have you ever read The Origin of Species?"
Laura was taken aback. "Do you mean Charles Darwin's treatise on evolution?" she asked, and Gray nodded. "Well, I've never actually read it, no."
"You should. It's the single greatest intellectual accomplishment in man's history." Laura arched her brow in surprise and looked at her colleagues around the table. They remained silent and deferential during Gray's lecture. "Genetic algorithms are a way of using Darwin's theories of evolution to program a computer to solve problems. In nature, the more fit an organism is, the more likely it is to pass on its genes. New organisms are evaluated harshly. The winners live and pass on their traits, the losers die. The system doesn't care what characteristics get handed down. By the mere act of survival, the organism has proved its superiority."
Gray seemed totally focused on her, concentrating on making himself understood as if her comprehension were of the utmost importance.
"We have instituted Darwin's rules of natural selection inside the computer," he said, his words spoken with such care that they possessed a gravity that fully captured Laura's attention. "Instead of organisms competing for survival, we have computer programs. Instead of passing genes on to their descendants, the programs that survive pass on the superior computer codes they use to solve problems. Their fitness is evaluated by how much those programs' solutions contribute to optimal computer performance. Programs that do poorly get eliminated from the gene pool. Programs that produce good results live on, and every so often they combine with other survivors. The new programs are sometimes defective, but other times they use novel approaches to arrive at truly unique solutions. That's why this computer has what no other has had before — brilliance. And the system works, Dr. Aldridge, only by strict adherence to the rules of natural selection. Only by pure survival of the fittest."
Laura felt, as always, out of her depth. "How in the world do you program a computer to operate like that?"
Margaret answered from her accustomed place to Gray's left. "Whenever there's a new problem, the computer mixes up a huge population of what we call 'chromosomes.' They're just tiny strings of connection sets — computer programs — that take data as their input and output a guess as to what the answer is. Genetic algorithms manage the whole process. The algorithm's selection operator chooses which chromosomes are most fit and mates them. That forms new and generally more sophisticated offspring programs that are then evaluated for mating on the basis of their fitness, et cetera, et cetera."
Dorothy picked up the thread. "That's how we think a virus might have naturally arisen. You see, it's important to preserve the diversity of the population. The genetic algorithm has to maintain a fine balance. If the programs get too inbred, they don't explore new problem-solving avenues. But if they mutate too often, it screws up the gradual improvement in the programming that natural selection gives you. So every once in a while, the genetic algorithm's mutation operator injects a random variable into the mix during the mating process. A different twist in the program's logic. I think what happened is that a mutant strain of highly capable chromosomes has developed and is at work in the system."
Laura looked from face to face around the table. This was all so mundane to them, but it sounded impossibly complex to her.
"Shouldn't the antiviral program catch a mutant virus like that?" she asked. "I mean, if it's causing all those errors…?"
Gray answered. "The phase-one, two, and three don't just rid the system of foreign viruses, they're also the weapon used by the selection operator to cull the programs that are less fit. They are the enforcers of the strict laws of Darwinism. Two programs compete against each other, and the antiviral programs await the loser. Those are the rules of the system, Dr. Aldridge. That is the law."
Gray seemed to wait for Laura to respond. When she didn't, he continued. "It's harsh justice, I know, but you've seen what it has accomplished. We don't interfere with the selection process," he stood making the point now to his still-silent department heads. "We can't. That's an unwavering rule."
"So why are you trying to manually load the phase-two?" Laura asked. "Isn't that interfering with the selection process?"
Several heads shook all at once, but it was Dorothy who replied. "The programs are hard to kill. They're meant to be robust, so they repel or flee from any reprogramming signals that might cause damage to their codes. Occasionally some are so good at survival that they avoid termination by the phase-one. They then become a 'virus.' And since this is a computer, those outbreaks can occur in just nanoseconds. That's why I programmed the phase-one to load the phase-two or — three automatically if the one's not up to the job."
"Then why are you doing all this manually?" Laura persisted.
"Because when the phase-one can't identify and localize the virus for a quick kill by a clipped version of the phase-two, we've got to free up enough space to load the entire program. The full version of an antiviral program scans, analyzes, identifies, tracks, localizes, destroys, repairs, and reports. Each step is a massive program that itself evolved through natural selection, combined by mating, and survived by out-competing opponents. Luckily, we've only had one outbreak that required a full loading of the most capable of the antiviral programs."
"The Hong Kong flu, or whatever you called it?" Laura asked.
Dorothy nodded, then fell silent. The mere mention of the episode seemed to cast a pall over the table.
Gray resumed the lecture. "In the case of the Hong Kong 1085, we had to go to the phase-three. Since the kill time was going to be extended, we had to manually off-load operations just like we're trying to do today." The strain showed on his face as he continued. "You see, the phase-one, — two, and — three are each completely different. They evolved independently. The phase-three was the 'winner' of the competition. It's the best killer by far. But we kept all three because the more capable a killer they are, the more damage they cause in the process. The phase-three is like chemotherapy. We used it only as a last resort." He looked now around the table. "We were losing the system." Everyone remained quiet. "The phase-three doesn't tiptoe around like the phase-one, or even the — two. When the phase-three goes in, it goes in hard."
"Slash and burn," Margaret said with distaste.
"It saved the system!" Dorothy objected testily.
"It killed the virus," Gray interjected, quieting the incipient debate. "We had to shut down for several days to work around the damage. It was touch-and-go, but the computer was able to relearn most of what it had lost by analogy to the undamaged connections. But if we'd lost much more code, or if what we did lose had been of a more critical nature, the system might've been unrecoverable."
The team remained quiet in the moments that followed. Most had sober looks on their faces.
"But the Hong Kong virus was man-made, right?" Laura asked.
"Yes," Gray said, "but it doesn't matter. A virus is a virus whether it's written by a hacker or it arose naturally from mutations while inside the system. In the case of the latter, it's a definition thing. It becomes a 'virus' when it loses its competition with a fitter program but then eludes or resists termination by the genetic algorithm."
"We call it the 'flora and fauna' of the system," Dorothy said.
"The 'wildlife,'" Filatov contributed.
"The point is," Gray said, "the connections are all constantly evolving. The amount of information contained in the computer's main code was initially equivalent to the amount of data stored in the DNA of a small rodent. It has grown, however, to a complexity approaching that of the human brain. And the viruses running loose in the system have evolved right along with it. The information content of their code used to be roughly the same as biological viruses' DNA. Now, some have codes similar in information content to the DNA of insects."
"They're ingenious," Margaret said. "Some of the naturally evolving viruses are 'nesters.' They infest whole racks in the nitrogen pools. When one member of the community is attacked, it sends out an alarm and the rest flee. And then other viruses are loners and scurry when attacked."
"We build traps for the slippery ones," an excited Dorothy chimed in. "We analyze their habitat preferences and lure them onto a board that has been inoculated with a customized antiviral program. The nesters we try to leave alone. They tend to max out at three or four columns of infestation. If we go in there too quickly, there's a chance they'll scatter all over the place, and then we're talking, like, massive infection. Instead, every so often we'll just quietly unplug the columns and reinitialize their connections."
Laura was dumbfounded. Computer operations in her mind were crisp and clear and above all else orderly. What they were describing were none of those things.
"The viruses are mostly benign, surprisingly," Gray picked up. "Many are predators that feed off other viruses."
"Like the Venus flytrap!" Dorothy said, smiling with evident enjoyment at the discussion of her specialty. "It was the coolest one we ever found. We didn't even know we had a problem. Our error rate was actually falling. But the virus grew voracious, eating its way through the system's flora and ultimately its fauna over a two-week period until it got to be too large. You see, it didn't delete the other viruses, it added them to its own code. By the end of the second week, it had grown like a tumor to a full percentage point of our total capacity, and excising it was tricky. If we broke it up, we potentially released all the viruses that had stuck to it. That would've wreaked havoc on the system." Dorothy heaved a clearly audible sigh. "I still think we could've caught it. We could have lured it into one of the [unclear] processors and shut it in there."
"And then what?" Filatov asked in an incredulous tone.
"Should we have rehabilitated your 'flytrap' so it could become a responsible member of our community? Used an entire Cray or a connection machine as an aquarium for your pet piranha?"
Dorothy looked up, shrugging. "But it was beautiful."
A silence descended on the room.
"What we're saying, Dr. Aldridge," Gray resumed, "is that the computer is an ecosystem. Dorothy has to maintain a balance. She can't thin the predators without risking a massive infection by their prey. The computer even has a symbiosis of sorts with its viruses, just as humans have coevolved with certain biological microorganisms. Today, without the benign bacteria in your stomach that aid in digestion, you'd die. In return for humans providing a host, those bacteria reproduce just rapidly enough to maintain healthy digestion. It's good for them, and it's good for us. If however, there's a perforation of the stomach wall, then all bets are off. The normally benign bacteria assume the host is a goner and begin to reproduce massively. That way, some of them may make it into a new host and perpetuate the species. But that massive infection is almost certain to kill the old host. That's why stomach wounds are so dangerous, and it's the same danger we deal with in the computer. We have to be careful to avoid stampeding the wildlife."
Laura again checked her colleagues around the table. They all wore serious looks on their faces.
"But all of those concerns," Gray said slowly in the stillness, "bow to one immutable law. The fittest program must survive. The less fit program dies. That's Darwin's law… and mine."
After lunch, Laura continued the analysis in her office.
"If you think I know why the phase-two didn't load, you obviously don't know anything about me," Laura typed and hit Enter.
Laura stared at the line. She was unnerved by the sudden shift in the discussion. But she was even more thrown by the fact the computer was correct. It was her favorite movie.
