Metamagical Themas, page 66
SANDY: But you wouldn't call it "simulated" sadness, would you?
PAT: No, of course not. I think it's real.
SANDY: It's hard to avoid use of such teleological or mentalistic terms. I believe they're quite justified, although they shouldn't be carried too far. They simply don't have the same richness of meaning when applied to present-day chess programs as when applied to people.
CHRIS: I still can't see that intelligence has to involve emotions. Why couldn't you imagine an intelligence that simply calculates and has no feelings?
SANDY: A couple of answers here. Number one, any intelligence has to have motivations. It's simply not the case, whatever many people may think, that machines could think any more "objectively" than people do. Machines, when they look at a scene, will have to focus and filter that scene down into some preconceived categories, just as a person does. And that means seeing some things and missing others. It means giving more weight to some things than to others. This happens on every level of processing.
PAT: I'm not sure I'm following you.
SANDY: Take me right now, for instance. You might think I'm just making some intellectual points, and I wouldn't need emotions to do that. But what makes me care about these points? Just now-why did I stress the work "care" so heavily? Because I'm emotionally involved in this conversation! People talk to each other out of conviction-not out of hollow, mechanical reflexes. Even the most intellectual conversation is driven by underlying passions. There's an emotional undercurrent to every conversation-it's the fact that the speakers want to be listened to, understood, and respected for what they are saying.
PAT: It sounds to me as if all you're saying is that people need to be interested in what they're saying, otherwise a conversation dies.
SANDY: Right! I wouldn't bother to talk to anyone if I weren't motivated by interest. And "interest" is just another name for a whole constellation of subconscious biases. When I talk, all my biases work together, and what you perceive on the surface level is my personality, my style. But that style arises from an immense number of tiny priorities, biases, leanings. When you add up a million of them interacting together, you get something that amounts to a lot of desires. It just all adds up! And that brings me to the other answer to Chris' question about feelingless calculation. Sure, that exists-in a cash register, a pocket calculator. I'd say it's even true of all today's computer programs. But eventually, when you put enough feelingless calculations together in a huge coordinated organization, you'll get something that has properties on another level. You can see it in fact, you have to see it-not as a bunch of little calculations but as a system of tendencies and desires and beliefs and so on. When things get complicated enough, you're forced to change your level of description. To some extent that's already happening, which is why we use words such as "want", "think", "try", and "hope" to describe chess programs and other attempts at mechanical thought. Dennett calls that kind of level-switch by the observer "adopting the intentional stance". The really interesting things in AI will only begin to happen, I'd guess, when the program itself adopts the intentional stance toward itself!
CHRIS: That would be a very strange sort of level-crossing feedback loop.
SANDY: It certainly would. When a program looks at itself from the outside, as it were, and tries to figure out why it acted the way it did, then I'll start to think that there's someone in there, doing the looking.
PAT: You mean an "I"? A self?
SANDY: Yes, something like that. A soul, even-although not in any religious sense. Of course, it's highly premature for anyone to adopt the intentional stance (in the full force of the term) with respect to today's programs. At least that's my opinion.
CHRIS: For me an important related question is: To what extent is it valid to adopt the intentional stance toward beings other than humans?
PAT: I would certainly adopt the intentional stance toward mammals.
SANDY: I vote for that.
CHRIS: Now that's interesting. How can that be, Sandy? Surely you wouldn't clairrr that a dog or cat can pass the Turing Test? Yet don't you maintain the Turing Test is the only way to test for the presence of consciousness? How can you have these beliefs simultaneously?
SANDY: Hmm ... All right. I guess that my argument is really just that the Turing Test works only above a certain level of consciousness. I'm perfectly willing to grant that there can be thinking beings that could fail at the Turing Test-but the main point that I've been arguing for is that anything that passes it would be a genuinely conscious, thinking being.
PAT: How can you think of a computer as a conscious being? I apologize if what I'm going to say sounds like a stereotype, but when I think of conscious beings, I just can't connect that thought with machines. To me, consciousness is connected with soft, warm bodies, silly though it may sound.
CHRIS: That does sound odd, coming from a biologist. Don't you deal with life so much in terms of chemistry and physics that all magic seems to vanish?
PAT: Not really. Sometimes the chemistry and physics simply increase the feeling that there's something magical going on down there! Anyway, I can't always integrate my scientific knowledge with my gut feelings. CHRIS: I guess I share that trait.
PAT: So how do you deal with rigid preconceptions like mine?
SANDY: I'd try to dig down under the surface of your concept of "machine" and get at the intuitive connotations that lurk there, out of sight but deeply influencing your opinions. I think we all have a holdover image from the Industrial Revolution that sees machines as clunky iron contraptions gawkily moving under the power of some loudly chugging engine. Possibly that's even how the computer inventor Charles Babbage saw people! After all, he called his magnificent many-geared computer the "Analytical Engine".
PAT: Well, I certainly don't think people are just fancy steam shovels or electric can openers. There's something about people, something that that-they've got a sort of flame inside them, something alive, something that flickers unpredictably, wavering, uncertain-but something creative !
SANDY: Great! That's just the sort of thing I wanted to hear. It's very human to think that way. Your flame image makes me think of candles, of fires, of vast thunderstorms with lightning dancing all over the sky in crazy, tumultuous patterns. But do you realize that just that kind of thing is visible on a computer's console? The flickering lights form amazing chaotic sparkling patterns. It's such a far cry from heaps of lifeless, clanking metal! It is flamelike, by God! Why don't you let the word "machine" conjure up images of dancing patterns of light rather than of giant steam shovels?
CHRIS: That's a beautiful image, Sandy. It does tend to change my sense of mechanism from being matter-oriented to being pattern-oriented. It makes me try to visualize the thoughts in my mind-these thoughts right now, even!-as a huge spray of tiny pulses flickering in my brain.
SANDY: That's quite a poetic self-portrait for a mere spray of flickers to have come up with!
CHRIS: Thank you. But still, I'm not totally convinced that a machine is all that I am. I admit, my concept of machines probably does suffer from anachronistic subconscious flavors, but I'm afraid I can't change such a deeply rooted sense in a flash.
SANDY: At least you sound open-minded. And to tell the truth, part of me sympathizes with the way you and Pat view machines. Part of me balks at calling myself a machine. It is a .bizarre thought that a feeling being like you or me might emerge from mere circuitry. Do I surprise you?
CHRIS: You certainly surprise me. So, tell us-do you believe in the idea of an intelligent computer, or don't you?
SANDY: It all depends on what you mean. We've all heard the question "Can computers think?" There are several possible interpretations of this (aside from the many interpretations of the word "think"). They revolve around different meanings of the words "can" and "computer".
PAT: Back to word games again ...
SANDY: I'm sorry, but that's unavoidable. First of all, the question might mean, "Does some present-day computer think, right now?" To this I would immediately answer with a loud no. Then it could be taken to mean, "Could some present-day computer, if suitably 'programmed, potentially think?" That would be more like it, but I would still answer, "Probably not". The real difficulty hinges on the word "computer". The way I see it, "computer" calls up an image of just what I described earlier: an air-conditioned room with cold rectangular metal boxes in it. But I suspect that with increasing public familiarity with computers and continued progress in computer architecture, that vision will eventually become outmoded.
PAT: Don't you think computers as we know them will be around for a while?
SANDY: Sure, there will have to be computers in today's image around for a long time, but advanced computers-maybe no longer called "computers"-will evolve and become quite different. Probably, as with living organisms, there will be many branchings in the evolutionary tree. There will be computers for business, computers for schoolkids, computers for scientific calculations, computers for systems research, computers for simulation, computers for rockets going into space, and so on. Finally, there will be computers for the study of intelligence. It's really only these last that I'm thinking of-the ones with the maximum flexibility, the ones that people are deliberately attempting to make smart. I see no reason that these will stay fixed in the traditional image. They probably will soon acquire as standard features some rudimentary sensory systems-mostly for vision and hearing, at first. They will need to be able to move around, to explore. They will have to be physically flexible. In short, they will have to become more animal-like, more self-reliant.
CHRIS: It makes me think of the robots R2D2 and C3PO in the movie Star Wars.
SANDY: Not me! In fact, I don't think of anything remotely like them when I visualize intelligent machines. They are too silly, too much the product of a film designer's imagination. Not that I have a clear vision of my own. But I think it's necessary, if people are realistically going to try to imagine an artificial intelligence, to go beyond the limited, hard-edged picture of computers that comes from exposure to what we have today. The only thing all machines will always have in common is their underlying mechanicalness. That may sound cold and inflexible, but then just think -what could be more mechanical, in a wonderful way, than the workings of the DNA and proteins and organelles in our cells?
PAT: To me, what goes on inside cells has a "wet", "slippery" feel to it, and what goes on inside machines is dry and rigid. It's connected with the fact that computers don't make mistakes, that computers do only what you tell them to do. Or at least that's my image of computers.
SANDY: Funny-a minute ago, your image was of a flame, and now it's of something wet and slippery. Isn't it marvellous, how contradictory we can be?
PAT: I don't need your sarcasm.
SANDY: No, no, I'm not being sarcastic-I really do think it's marvellous. PAT: It's just an example of the human mind's slippery nature-mine, in this case.
SANDY: True. But your image of computers is stuck in a rut. Computers certainly can make mistakes-and I don't mean on the hardware level. Think of any present-day computer predicting the weather. It can make wrong predictions, even though its program runs flawlessly.
PAT: But that's only because you've fed it the wrong data.
SANDY: Not so. It's because weather prediction is too complex. Any such program has to make do with a limited amount of data-entirely correct data-and extrapolate from there. Sometimes it will make wrong predictions. It's no different from a farmer gazing at the clouds and saying, "I reckon we'll get a little snow tonight." In our heads, we make models of things and use those models to guess how the world will behave. We have to make do with our models, however inaccurate they may be, or evolution will prune us out ruthlessly-we'll fall off a cliff or something. And for intelligent computers, it'll be the same. It's just that human designers will speed up the evolutionary process by aiming explicitly at the goal of creating intelligence, which is something nature just stumbled on.
PAT: So you think computers will be making fewer mistakes as they get smarter?
SANDY: Actually, just the other way around! The smarter they get, the more they'll be in a position to tackle messy real-life domains, so they'll be more and more likely to have inaccurate models. To me, mistake-making is a sign of high intelligence!
PAT: Wow-you throw me sometimes!
SANDY: I guess I'm a strange sort of advocate for machine intelligence. To some degree I straddle the fence. I think that machines won't really be intelligent in a humanlike way until they have something like your biological wetness or slipperiness to them. I don't mean literally wet-the slipperiness could be in the software. But biological-seeming or not, intelligent machines will in any case be machines. We will have designed them, built them-or grown them! We'll understand how they work-at least in some sense. Possibly no one person will really understand them, but collectively we will know how they work.
PAT: It sounds like you want to have your cake and eat it too. I mean, you want to have people able to build intelligent machines and yet at the same time have some of the mystery of mind remain.
SANDY: You're absolutely right-and I think that's what will happen. When real artificial intelligence comes
PAT: Now there's a nice contradiction in terms!
SANDY: Touche! Well, anyway, when it comes, it will be mechanical and yet at the same time organic. It will have that same astonishing flexibility that we see in life's mechanisms. And when I say mechanisms, I mean mechanisms. DNA and enzymes and so on really are mechanical and rigid and reliable. Wouldn't you agree, Pat?
PAT: Sure! But when they work together, a lot of unexpected things happen. There are so many complexities and rich modes of behavior that all that mechanicalness adds up to something very fluid.
SANDY: For me, it's an almost unimaginable transition from the mechanical level of molecules to the living level of cells. But it's that exposure to biology that convinces me that people are machines. That thought makes me uncomfortable in some ways, but in other ways it is exhilarating.
CHRIS: I have one nagging question ... If people are machines, how come it's so hard to convince them of the fact? Surely a machine ought to be able to recognize its own machinehood!
SANDY: It's an interesting question. You have to allow for emotional factors here. To be told you're a machine is, in a way, to be told that you're nothing more than your physical parts, and it brings you face to face with your own vulnerability, destructibility, and, ultimately, your mortality. That's something nobody finds easy to face. But beyond this emotional objection, to see yourself as a machine, you have to "unadopt" the intentional stance you've grown up taking toward yourself-you have to jump all the way from the level where the complex lifelike activities take place to the bottom-most mechanical level where ribosomes chug along RNA strands, for instance. But there are so many intermediate layers that .they act as a shield, and the mechanical quality way down there becomes almost invisible. I think that when intelligent machines come around, that's how they will seem to us-and to themselves! Their mechanicalness will be buried so deep that they'll seem to be alive and conscious just as we seem alive and conscious ...
CHRIS: You're baiting me! But I'm not going to bite.
PAT: I once heard a funny idea about what will happen when we eventually have intelligent machines. When we try to implant that intelligence into devices we'd like to control, their behavior won't be so predictable. SANDY: They'll have a quirky little "flame" inside, maybe? PAT: Maybe.
CHRIS: And what's so funny about that?
PAT:. Well, think of military missiles. The more sophisticated their target-tracking computers get, according to this idea, the less predictably they will function. Eventually, you'll have missiles that will decide they are pacifists and will turn around and go home and land quietly without blowing up. We could even have "smart bullets" that turn around in midflight because they don't want to commit suicide!
SANDY: What a nice vision!
CHRIS: I'm very skeptical about all this. Still, Sandy, I'd like to hear your predictions about when intelligent machines will come to be.
SANDY: It won't be for a long time, probably, that we'll see anything remotely resembling the level of human intelligence. It rests on too awesomely complicated a substrate-the brain-for us to be able to duplicate it in the foreseeable future. Anyhow, that's my opinion.
PAT: Do you think a program will ever pass the Turing Test?
SANDY: That's a pretty hard question. I guess there are various degrees of passing such a test, when you come down to it. It's not black and white. First of all, it depends on who the interrogator is. A simpleton might be totally taken in by some programs today. But secondly, it depends on how deeply you are allowed to probe.
PAT: You could have a range of Turing Tests-one-minute versions, five minute versions, hour-long versions, and so forth. Wouldn't it be interesting if some official organization sponsored a periodic competition, like the annual computer-chess championships, for programs to try to pass the Turing Test?
CHRIS: The program that lasted the longest against some panel of distinguished judges would be the winner. Perhaps there could be a big prize for the first program that fools a famous judge for, say, ten minutes.
PAT: A prize for the program, or for its author?
CHRIS: -For the program, of course!
PAT: That's ridiculous! What would a program do with a prize?
CHRIS: Come now, Pat. If a program's human enough to fool the judges, don't you think it's human enough to enjoy the prize? That's precisely the threshold where it, rather than its creators, deserves the credit, and the rewards. Wouldn't you agree?
PAT: Yeah, yeah-especially if the prize is an evening out on the town, dancing with the interrogators!
