Isaac Asimov imagined his robots in the shape of humans so that they would fit well into human society and be as useful as possible. When designing a robot, the external shape is important to get right, but it is not the most important aspect of the design. The most critical element of a robot is its brain, sometimes called an Artificial Intelligence or AI. When designing an AI, a thinking being, the crucial question is: what should be its personal goals? Should a robot AI simply follow orders or should it have default rules already built-in?

The Three Laws of Robotics

1. A robot may not injure a human being, or, through inaction, allow a human being to come to harm.
2. A robot must obey orders given it by human beings, except where such orders would conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

These laws look quite good upon first reading; they seem to be a reasonable guide for an intelligent being toward behaviour that a normal person would find acceptable, if not exemplary! Of course, it is easy to assume that a robot would interpret the laws the same way a human being would, but there is a problem. A human will have human goals and a variety of other mental imperatives such as breathing, eating and drinking, securing shelter and safety, finding love, building a family, etc. Asimov’s robots do not have any of these goals or imperatives; their only goal is to obey the 3 Laws. This will lead to many problems, because the robot will interpret the laws differently from a human since it has different goals. Laws and goals are different things, but the two are mixed together in Isaac Asimov’s 3 Laws of Robotics. The dictionary definitions of “law” and “goal” are:

LAW n. rule of action or procedure commanding or forbidding certain actions.
GOAL n. object of ambition or effort.

There are also other differences between human and robot interpretations. A 3 Laws robot will have very different experiences of the world in comparison to a human. How can we expect it to obey laws like a human without the experience of growing up through childhood and into adulthood as a human? We can’t expect that; it won’t happen automatically. An even worse problem is that a 3 Laws robot would have none of the moral framework of a human adult; none of the appreciation for beauty or the love of family and community, nor the wisdom to determine right from wrong. According to a 3 Laws robot, the “right thing to do” has nothing to do with wisdom; the right thing to do is forever fixed as “obey the 3 Laws.”

Human-made laws work well in human society because people have reasons and motivations to follow the rules. There are many motives, both positive and negative. A positive example is: “If I follow the law, then society will operate more smoothly, and we will all be better off.” The human motivation here is altruism, to benefit others, and also selfishness because benefiting society indirectly benefits the self. “To avoid punishment for breaking laws,” is an obvious example of the negative. But a 3 Laws robot shares none of these human motivations. All an Asimovian robot brain knows is to follow the 3 Laws. If the 3 Laws do not seem to apply, what happens then?

If human beings encounter a completely new problem, then they will do various things to try and solve it. Problem solving requires intelligence, and the more intelligent humans are the better they can solve new and unique problems. If robots are to obey the 3 Laws like an intelligent human they will also have to be intelligent. Robots would have to be at least as intelligent as humans to understand how we want the laws obeyed. But, as we have already seen, 3 Laws robots have different motivations from humans; their motivation is simply to obey the 3 Laws. Therefore a 3 Laws robot will always obey the laws differently from a human. A ready example is when a human being wants to do something where there is a risk of harm, such as crossing a street. What would a robot do when observing that situation? It might restrain the person from crossing a street, even if the risk of harm was low. And there are other, even more troubling, problems.

The goal of “obedience to a set of laws” does not account for exceptions to the laws, or future changes to the laws. Even worse from the point of view of us poor humans: what if the laws are flawed? Or what if we made a mistake and they are flat-out wrong? There will be many new situations occurring in the future, and the laws we come up with today will not be perfect tomorrow – they will need to be modified. But laws are notoriously slow to change in human society, and there is no reason to suspect that robots will be upgraded with new sets of laws fast enough to keep up with new situations. This is true even today with ordinary computer software that is quickly and automatically updated online to fix bugs and “vulnerabilities.” Despite our greatest efforts, new invasions of internet worms and software viruses still cause havoc before the software can be patched to handle a new situation. So… is there no hope for these great-sounding 3 Laws? Would a truly intelligent robot AI find creative ways to make the laws work the way they were intended?

Taking the First Law as an example, a truly intelligent 3 Laws robot might find ways to creatively obey the law. The last part of the First Law states that a robot may not, through inaction, allow a human being to come to harm. This directly implies that, if a human being might come to harm, the robot must take action. It would not take long observing humans for an intelligent robot to conclude that there are many situations during daily life when a human might come to harm. Some of the more frequent accidents are auto collisions, slipping and falling, and back injuries due to lifting heavy objects. A 3 Laws robot would certainly want to take action to prevent these harmful events. A robot might not let you drive because of the risk of collision, or walk when the ground is wet because you might slip. Perhaps a robot would prevent you from getting out of bed – after all your body is quite heavy, and you might fall and hurt yourself. These examples may seem humorous or trivial, but you wouldn’t be laughing if they actually happened.

Asimov’s robotic laws were designed by a writer for literary purposes, and his stories were often interesting because the robots’ behaviour was not the behaviour expected of humans. Unexpected outcomes make for interesting story telling but would be very uncomfortable as a lifestyle, especially if it involved imprisonment by an overprotective 3 Laws robot. There are many reasons why creating an intelligent being with goals to “obey laws” will inevitably lead to failure; this article only covers a few of them. Using laws as goals cannot work, and even ten thousand laws are not enough. The failures would continue, in slightly different ways with different sets of laws, but for the same reasons as they fail for 3 Laws. Laws don’t prevent catastrophes, or make good behaviour certain. Only rational thinking beings, who have goals of working in cooperation with humans, could obey laws we specify; and then continue to obey the spirit of the laws when the original laws no longer make sense. Such beings will not be bound by laws because of a few lines of computer code, but because they are complex moral beings that can reason about the world and choose to do what is right.

For further information about alternatives to the 3 Laws, see http://www.singinst.org/intro/

One of the most persuasive themes in literature is that of the artificial servant. In his 1921 play “RUR,” Karel Capek named his artificial servants “robots,” from the Czech word robota, which roughly translates as “unwilling worker.” (It could also mean “someone who does boring work.”) That name has stuck with us ever since, and despite other names – android (human-shaped artificial being), cyborg (cybernetic organism, implying part machine, part living tissue), droid, and so on – robot is likely to be the definitive label for some decades to come.

Later, the Three Laws became the key to plot development in the robot stories, and as Asimov merged his robot stories with his Foundation stories and novels, the Three Laws played a significant role there, as well. Today, they are taken for granted – by science fiction writers and robot designers alike.

The Three Laws seem almost self-evident as guidelines for any variety of artificial servant – at least, servants capable of moving around or otherwise taking significant action. Thus, while a modern thermostat (a particularly simple kind of robot) has neither the complexity nor the power to save or harm a human, the onboard computer of a spaceship may. (Asimov was indignant that the HAL 9000 computer in Kubrick and Clarke’s 2001: A Space Odyssey failed to follow the Three Laws! Clarke and Kubrick, however, implied that a programming contradiction caused Hal to malfunction, and that was the root cause of Hal’s murderous behavior.)

A completely mobile metal servant in humanoid form, as envisioned by Asimov, is mythic. Robot servants of this type have existed at least since Homer (see footnote). Even before Asimov’s first robot compendium, Eando Binder (the writing team of Earl and Otto Binder) had written a story called “I, Robot,” later dramatized on the 1960’s television series The Outer Limits. But Asimov gave the robots an endearing personality, a feckless but determined sense of honor and dignity that has not been equaled since.

It’s not at all extreme to call Asimov the father of the modern robot, just as his character, Susan Calvin, is the fictional mother of the robots. Calvin, of course, creates the science of robotics – a word that Asimov coined, and which has gone into general usage today.

Some of the most famous robots of the screen show a selfless fealty to the Three Laws. Robby the Robot in Forbidden Planet (1956) is beyond any doubt an Asimovian robot. The terminators in James Cameron’s series of films are rather less so.

Ultimately, even Asimov himself maneuvered around the basic Three Laws. R. Daneel Olivaw, the robot colleague of detective Lije Bailey in The Caves of Steel and The Naked Sun, when faced with the extreme forces of human history, and some very difficult decisions, formulates the so-called Zeroth Law in Robots and Empire. Basically, the Zeroth law allows a robot to harm a human if it is for the expressed good of a greater number of humans, or humanity as a whole. This allows Daneel to become a behind-the-scenes manipulator of human history, a sort of one-robot Illuminati. But Asimov was well aware of the dire implications of the Zeroth Law, and was still developing Daneel’s story in the last robot/Foundation novels.

Gregory Benford, David Brin, and I used Daneel in our trilogy of novels set in Asimov’s Foundation universe. Daneel in these novels has become a tragic figure, neither human nor entirely robot, burdened by the weight of history. For the Zeroth law, of course, allows and even encourages robotic interpretation of circumstances, thereby implying – even requiring – that robots have the equivalent of a conscience.

To me, it seemed unlikely that Daneel would appreciate these burdens. But, as a robot, Daneel must serve…
Forever and ever.

Asimov was well aware of the myth of the Golem, an artificial warrior/protector made of clay and imbued with life in 16th Century Prague by Rabbi Löw. The Golem is instructed to protect the Jewish inhabitants of the ghetto against the depredations of anti-semitic mobs. But the Golem is too powerful and ultimately goes out of control. This is the classic version of the artificial servant story, and it is echoed in the extraordinary novel and myth of Frankenstein’s monster. However, Mary Shelley’s monster, assembled by a scientist from fresh corpses, is not intended to be a servant, but a new kind of human being, intellectually and physically a superman. The Monster’s failure is due to another kind of programming error – the failure of Frankenstein, his creator, to follow through with a moral education, to take responsibility for this new kind of “child.”

Asimov’s robots are often childlike, but their programming failures usually arise through inherent contradictions in the Three Laws, or unexpected mutations, such as the development of ESP. The robots themselves, by and large, remain innocent, like perpetual children.

As for their positronic brains, it’s clear that Asimov was simply looking for a new way of describing super-advanced and complicated computer circuitry. Positrons are anti-electrons. Circuits utilizing positrons would be exotic indeed. This is not to say they are impossible – just very advanced. Besides, the name sounds cool. Today, scientists are working with quantum computers that rely on the vagaries of the subatomic realm. Such computers can do computations in more than one “universe” at a time. Doubtless Asimov would have found that a very intriguing possibility.

What if a robot with a quantum logic brain could act in more than one world-line? What if, in one branching world-line, a robot must kill a human being, so that in all other world-lines, humanity will survive?

As you can see, the Three Laws can be resurrected and played with over and over again.

As always, Asimov endures, and the problems he presents to us in these stories are, in disguise, the moral problems of any thinking individual.

In the end, what intrigues us most about the robot stories and novels is the conviction that we are no better than the robots, and perhaps worse. Perhaps we are simply the servants of human history, and our discovery of the Zeroth Law led to our Fall from grace.

Are the Ten Commandments nothing more than a wordy expansion of the Three Laws?

—

Footnote: The Tin Woodsman in Baum’s The Wizard of Oz is actually a kind of Six Million Dollar Man, since he started out as a human. As he accidentally chopped off portions of his anatomy, they were replaced by metal parts. Along the way, he would have been a cyborg!

(Partial reprint. Originally published as two parts in IEEE Computer, December 1993)

With the death of Isaac Asimov on April 6, 1992, the world lost a prodigious imagination. Unlike fiction writers before him, who regarded robotics as something to be feared, Asimov saw a promising technological innovation to be exploited and managed. Indeed, Asimov’s stories are experiments with the enormous potential of information technology.

This article examines Asimov’s stories not as literature but as a gedankenexperiment – an exercise in thinking through the ramifications of a design. Asimov’s intent was to devise a set of rules that would provide reliable control over semi-autonomous machines. My goal is to determine whether such an achievement is likely or even possible in the real world. In the process, I focus on practical, legal, and ethical matters that may have short- or medium-term implications for practicing information technologists.

Robotics, a branch of engineering, is also a popular source of inspiration in science fiction literature; indeed, the term originated in that field. Many authors have written about robot behaviour and their interaction with humans, but in this company Isaac Asimov stands supreme. He entered the field early, and from 1940 to 1990 he dominated it. Most subsequent science fiction literature expressly or implicitly recognizes his Laws of Robotics.

Asimov described how, at the age of 20 he came to write robot stories:

“In the 1920’s science fiction was becoming a popular art form for the first time ….. and one of the stock plots …. was that of the invention of a robot …. Under the influence of the well-known deeds and ultimate fate of Frankenstein and Rossum, there seemed only one change to be rung on this plot - robots were created and destroyed their creator … I quickly grew tired of this dull hundred-times-told tale …. Knowledge has its dangers, yes, but is the response to be a retreat from knowledge? …. I began in 1940, to write robot stories of my own - but robot stories of a new variety …… My robots were machines designed by engineers, not pseudo-men created by blasphemers”¹,²

Asimov was not the first to conceive of well-engineered, non-threatening robots, but he pursued the theme with such enormous imagination and persistence that most of the ideas that have emerged in this branch of science fiction are identifiable with his stories.

To cope with the potential for robots to harm people, Asimov, in 1940, in conjunction with science fiction author and editor John W. Campbell, formulated the Laws of Robotics.³,⁴ He subjected all of his fictional robots to these laws by having them incorporated within the architecture of their (fictional) “platinum-iridium positronic brains”. The laws (see below) first appeared publicly in his fourth robot short story, “Runaround”⁵.

The 1940 Laws of Robotics

First Law: A robot may not injure a human being, or, through inaction, allow a human being to come to harm.

Second Law: A robot must obey orders given it by human beings, except where such orders would conflict with the First Law.

Third Law: A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

The laws quickly attracted – and have since retained – the attention of readers and other science fiction writers. Only two years later, another established writer, Lester Del Rey, referred to “the mandatory form that would force built-in unquestioning obedience from the robot”.⁶

As Asimov later wrote (with his characteristic clarity and lack of modesty), “Many writers of robot stories, without actually quoting the three laws, take them for granted, and expect the readers to do the same”.

Asimov’s fiction even influenced the origins of robotic engineering. “Engelberger, who built the first industrial robot, called Unimate, in 1958, attributes his long-standing fascination with robots to his reading of [Asimov’s] ‘I, Robot’ when he was a teenager”, and Engelberger later invited Asimov to write the foreword to his robotics manual.

The laws are simple and straightforward, and they embrace “the essential guiding principles of a good many of the world’s ethical systems.”⁷ They also appear to ensure the continued dominion of humans over robots, and to preclude the use of robots for evil purposes. In practice, however – meaning in Asimov’s numerous and highly imaginative stories – a variety of difficulties arise.

My purpose here is to determine whether or not Asimov’s fiction vindicates the laws he expounded. Does he successfully demonstrate that robotic technology can be applied in a responsible manner to potentially powerful, semi-autonomous and, in some sense intelligent machines? To reach a conclusion, we must examine many issues emerging from Asimov’s fiction.

History

The robot notion derives from two strands of thought, humanoids and automata. The notion of a humanoid (or human- like nonhuman) dates back to Pandora in The Iliad, 2,500 years ago and even further. Egyptian, Babylonian, and ultimately Sumerian legends fully 5,000 years old reflect the widespread image of the creation, with god- men breathing life into clay models. One variation on the theme is the idea of the golem, associated with the Prague ghetto of the sixteenth century. This clay model, when breathed into life, became a useful but destructive ally.

The golem was an important precursor to Mary Shelley’s Frankenstein: The Modern Prometheus (1818). This story combined the notion of the humanoid with the dangers of science (as suggested by the myth of Prometheus, who stole fire from the gods to give it to mortals). In addition to establishing a literary tradition and the genre of horror stories, Frankenstein also imbued humanoids with an aura of ill fate.

Automata, the second strand of thought, are literally “self- moving things” and have long interested mankind. Early models depended on levers and wheels, or on hydraulics. Clockwork technology enabled significant advances after the thirteenth century, and later steam and electro- mechanics were also applied. The primary purpose of automata was entertainment rather than employment as useful artifacts. Although many patterns were used, the human form always excited the greatest fascination. During the twentieth century, several new technologies moved automata into the utilitarian realm. Geduld and Gottesman⁸ and Frude² review the chronology of clay model, water clock, golem, homunculus, android, and cyborg that culminated in the contemporary concept of the robot.

The term robot derives from the Czech word robota, meaning forced work or compulsory service, or robotnik, meaning serf. It was first used by the Czech playwright Karel Çapek in 1918 in a short story and again in his 1921 play R. U. R., which stood for Rossum’s Universal Robots. Rossum, a fictional Englishman, used biological methods to invent and mass- produce “men” to serve humans. Eventually they rebelled, became the dominant race, and wiped out humanity. The play was soon well known in English- speaking countries.

Continue reading Roger Clarke’s Asimov’s Laws of Robotics

—

¹ I. Asimov, The Rest of the Robots (a collection of short stories originally published between 1941 and 1957), Grafton Books, London, 1968.

² N. Frude, The Robot Heritage, Century Publishing. London. 1984.

³ I. Asimov, I, Robot (a collection of short stories originally published between 1940 and 1950), Grafton Books, London, 1968.

⁴ I. Asimov, P.S. Warrick, and M.H. Greenberg, eds., Machines That Think, Holt, Rinehart. and Wilson, London. 1983.

⁵ I. Asimov, “Runaround” (originally published in 1942), reprinted in Reference 3, pp. 33- 51.

⁶ L. Del Rey, “Though Dreamers Die” (originally published in 1944). reprinted in Reference 4, pp. 153- 174.

⁷ I. Asimov, “Evidence” (originally published in 1946). reprinted in Reference 3. pp. 159- 182.

⁸ H.M. Geduld and R. Gottesman. eds.. Robots, Robots, Robots, New York Graphic Soc., Boston. 1978.

Asimov’s 3 Laws of Robotics may seem a decent set of guidelines for ensuring that future robots and AIs behave in satisfactory ways. But there are several problems that immediately emerge when we look deeper. For one, it’s not so straightforward to convert a set of statements into a mind that follows or believes in those statements. Two, semantic ambiguity means that without personally understanding the reasons for the laws and the original intent, a robot might misinterpret their meaning, leading to problems. Third, Asimov’s Laws ignore the possibility that a robot will acquire the ability to reprogram itself – an inevitable eventuality if intelligent robots are created. How can we confront these issues, as we move closer to the creation of a new intelligent species?

1. A robot may not injure a human being, or, through inaction, allow a human being to come to harm.
2. A robot must obey orders given it by human beings, except where such orders would conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

Asimov’s Laws are humanity’s early, off-the-cuff answer to the question, “How do we build robots that get along well in human society?” But they led to failure in his stories, and would lead to failure if implemented in real life. Depending on the physical and intellectual power of future robots, and their ability to improve these powers, these failures could range from spilt coffee to the total extermination of the human race.

Asimov’s Three Laws of Robotics describe a minimum set of rules that many humans would prefer that all robots follow. But the rules, in their conciseness, lack content. Words like “robot,” “orders,” “humans,” and “existence” are fuzzy semantic constructs whose meaning lies primarily in the minds that interpret them. Connecting such words with concrete classes of physical systems or processes is not as easy as it seems. Human beings are fortunate to have a rich underlying base of cognitive complexity that connects a few marks on paper (words) to entire classes of complex physical processes.

Often the Laws are mistakenly presented as sufficient to create a robot that is thoroughly safe and fits in well with society. But the Laws are vastly insufficient; they’re simply too simple. The only working examples of benevolence and harmony with society that we have are well-balanced humans – whose behavior is implemented by brains that are enormously complex. In order to create truly benevolent AIs, we will need to create designs with the motivation to absorb major portions of this complexity.

Consider the apparently simple action of sweeping a floor with a broom. To us, it seems straightforward, but the computations occurring are actually far more elaborate than anything performed by present-day computer software. Maintaining balance, scanning a room for dust, performing hand motions to manipulate the broom into the chosen positions – a team of roboticists might slave for decades and still be unable to duplicate such complex processes.

“Getting along well with others” may seem straightforward to us, but it would actually require a goal system of substantial complexity to replicate. Asimov’s Laws ignore the extensive cognitive complexity wrapped up in any intelligent social morality. At best, Asimov’s Laws are a first tiny step on a huge journey of satisfactory goal system design in robots. At worst, they will give pre-robotic human civilization a false sense of security and preparedness, discouraging further research in the relevant areas.

Part of the issue is that humans are heavily loaded with instincts, common sense, and rules of thumb that assist us in determining which actions are acceptable and constructive, to ourselves and to others. Creating robots that add value to human society will not be a matter of creating a “generic mind” and then “constraining” it with Asimovian laws, but creating sensible and benevolent minds from the get-go. Gandhi was not a nice guy because someone forced him to be that way, but because that was part of his inborn character. Future robots should be the same.

In creating future robots and AIs, we will need to plan for scenarios in which robots become smarter than humans, bodies far more effective at manipulating reality than humans, robots which manipulate their own source code, and so on. Pessimists might throw up their arms in despair and say, “At that point, there’s nothing you can do. There is no way a human programmer can code an AI with certain goals and expect it to stay that way. You can’t control something smarter than you are.”

Yet, the very same humans that say these things behave in ways derived from their creator – evolution. Present-day human goal systems are derivatives of our original evolutionary goals. We may not be as uncivilized, barbaric, or narrow-minded as our ancestors, but our goals still firmly derive from evolutionary programming. Even if we decided to completely “throw out” our original goals, the choices we made about throwing out those goals would still somehow derive from the root goals!

Intelligent robots will be the same. A benevolent robot modifying its own programming will not randomly turn to selfishness, unless a cognitive predisposition in that direction is already present. A selfish robot modifying its own programming will not randomly turn to benevolence either. Although we may expect any sort of robot to start modifying itself to become more humanlike, this is pure science fiction without any scientific foundation. Any self-modifying robot will be an n-th order derivative of its initial conditions, and it will be up to us to ensure that those initial conditions contain enough seed complexity to develop into something we can be proud of.

Isaac Asimov created the heroine Susan Calvin, a legendary robot designer and trouble-shooter. Susan foresaw the eventual outcome of robots spreading throughout the world. She imagined that robots with the 3 Laws would become intellectually superior to humans and would rule over them, protecting them from harm. Why did she think this would happen? Let’s look closely at the 3 Laws and find out.

The Three Laws of Robotics

1. A robot may not injure a human being, or, through inaction, allow a human being to come to harm.
2. A robot must obey orders given it by human beings, except where such orders would conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

The Three Laws of Robotics is a hierarchy of commands; each law defers to the one above. The Second Law takes precedence only when the First Law does not apply. Take the example of asking a robot to bring you a spoon. This would not harm a human being, so the First Law would not apply and the robot must obey the order. The Third Law is used only when both the Second and First Laws do not apply. A robot attending to its own periodic maintenance without being told would be an application of the Third Law. With the First Law at the top of the hierarchy, circumstances where a human being might come to harm, such as crossing a busy freeway, are called First Law Situations. Any time a First Law Situation arises, the other laws are swept aside. Overwhelming priority is given to the First Law – as often depicted by Asimov, it overrules all the others.

In Asimov’s earliest stories, the robots are written as passively obeying the Three Laws as well as they are able. It is likely that most people who think the Three Laws are a good idea are thinking of passive obedience. By “passive” I mean not interfering in the day-to-day life of humans, other than to act as rescuers or protectors in situations where humans are in immediate danger of harm. But what would happen if a robot began to play more than a passive role in implementing the First Law? What if a robot actively attempted to obey the First Law to its maximum capability? Asimov starts answering this question in the last two stories of his book I, Robot, titled “Evidence” and “The Evitable Conflict,” where robots do not simply obey orders, but take an active role in world events.

A robot brain is a type of Artificial Intelligence, or “AI.” When discussing robots and AIs, many people use the words interchangeably to mean the same thing. There were several directions Asimov didn’t go with his robot AIs, such as recursive self-enhancement. Recursive self-enhancement occurs when an AI improves its own intelligence, and then repeats the process – but this time using more intelligence – and repeating again and again, resulting in a mountainous intellect. Even though Asimov didn’t write much about recursive self-enhancement, his robot AIs still had imagination. If a robot were to imagine itself with greater capability, then it would be straightforward for it to conclude that it would have greater ability to obey the First Law. This would certainly lead the robot to want to improve its mental and physical abilities, and perhaps even its social abilities, with the intent of increasing its political power and influence. A robot improving itself in this way would obtain an increasing spiral of capability completely overpowering that of humans, all to better obey the First Law and protect humans from harm.

It is easy to imagine that successfully creating a 3 Laws robot, an intelligent and willing servant, would eventually rob us of the opportunity of failing. Asimov’s phrase, “allow a human being to come to harm,” if implemented fully, would turn humanity into a clutch of coddled infants, perpetually protected from harm, both physical and mental. If this sounds like a disaster to you, then you are not alone. Almost no one would want so much of their initiative usurped in the name of “protection against harm.” Our evolutionary history is full of risk and hardship, including the ultimate harm: death. Humans have been formed in the crucible of struggle, clawing our way up to the top of the heap, and we are now on the cusp of being able to forge our own destiny. Now is not the time to relinquish our volition – our freedom to decide for ourselves!

Asimov repeated the theme of robots ruling over humans in later novels when he had his robots invent the Zeroth Law: “No robot may injure humanity, or through inaction allow humanity to come to harm.” Through the application of the Zeroth law, the robots influence the progress of humanity from the shadows, unable to step out and take open control because it would harm humanity’s pride. This makes for great story-telling, but we in the real world shouldn’t settle for merely a good story; there are better options.

Partnership is a far better option. Not an owned servant, but a friend and partner. Not a being controlling humanity through greater intellect, but allowing us to build our own destiny and offering help with the process. “Partnership” means different things to different people, but several principles are broadly held. Partnership includes:

• Freedom of action for both parties. “Freedom” implies that if we create a thinking, conscious being, it must not relate to us as a slave to a master, as 3 Laws robots would.
  
• Mutual trust and respect, requiring a shared moral frame of reference. Trust and respect are impossible to implement via simplistic laws. Human morals are highly complex and an AI should have the ability to understand morality at its deepest levels. Laws, no matter how many in number, cannot contain the complexity of our morals. Over time, morals gradually change, and our partners should have the ability to change with us.
  
• Reciprocal accountability, meaning each party involved is accountable to the other for their actions. For example, in a voluntary partnership the partners agree to do various things for each other and are accountable to each other for delivering on their promises.
  
• Transparency, and no hidden agendas. True partnership is an honest and open cooperative agreement. Neither party can be “in it for what they can get” but rather each is committed to work for the benefit of all.
  
• Long-term commitment, providing stability for planning. A partnership is not something that lasts an hour or a day, it is a much more enduring arrangement. From such a secure base of continuous mutual support, large projects can be started, and great things accomplished – things that would be difficult or impossible to tackle alone.
  
• Working together as allies rather than opponents. Partners are capable and willing to disagree without being constrained to avoid hurting each other’s feelings. A disagreement does not signal the termination of a partnership. There will be many activities that humans will want to do on their own, without the participation of non-human partners.
  
• Building the capacity of one’s partners. The pursuit of mutual benefit rather than one-sided benefit is the aim. From this point of view to build one’s partner’s capacities is also to build one’s own capacities.

The type of partnership illustrated above is the kind entered into by adults, not children. Shall we create 3 Laws robots and be treated like infants, as Susan Calvin predicted, or do we wish to be treated as grown-ups? Do we want to be told what to do by robot-parents or freely decide for ourselves in cooperation with robot-partners? Are we ready to surrender our volition – our power to decide for ourselves? There is nothing wrong with asking advice from an intelligent AI when making important decisions, but we must not hand over control of our lives to robots. Having a good advisor is one thing, but letting someone else decide for us is quite another.

The First Law of Robotics trumps all the other laws. One law: A robot may not injure a human being, or, through inaction, allow a human being to come to harm. Shall we be ruled by this? We should dread the day.

—

There are better ways to construct AIs than by making them adhere to a static hierarchy of laws, however appealing they might appear to be. For thoughtful alternatives, see: http://www.singinst.org/intro/.

That the Three Laws are insufficient to guarantee robot behavior should be obvious to anyone who has read Asimov’s stories. Usually the main plot is about misbehaving robots and the mystery is why – rather than being “whodunits” they are “howthinkits.” But how complex do the rules of robot behavior have to be before we can consider them safe?

To many people “computers just do what they are told” is an article of faith. As personal computers become more widespread, the fallacy of this statement becomes apparent: computers crash, refuse to print documents, update their software and occasionally exhibit remarkably strange behavior. Often these actions are the result of complex interactions between pre-existing software modules, where human ingenuity couldn’t predict the outcome of the particular combination in a particular computer at a given time. But despite this, people often seem to think that creating artificial intelligence would produce something that one could give a set of rules and have it follow slavishly.

In Asimov’s stories this is the case, and the result is chaos. In reality things will be even worse: beside ambiguities in the rules and how they are to be applied, there will be errors in cognition, perception and execution of actions. And of course low-level crosstalk and software bugs too. And this is just in the case of an ordinarily intelligent machine. The self-enhancing AIs envisioned by the Singularity Institute will have far more degrees of freedom to train against reality (and hence able to get wrong), and the number of potential non-obvious interactions increases exponentially.

So what to do about it? The idea I really like about Friendly AI is the attempt to formulate a goal architecture that is robust. If something goes wrong the system tries to adjust itself to make things better. It is no guarantee that it works, but experience shows that some systems are far less brittle than others.

Real intelligence exhibits several important traits: it interacts with its world, it is able to learn new behaviors (or unlearn old), and it can solve new problems using earlier information, heuristics and strategies. The learning aspect enables us to speak about the ethics of an AI program: how does it live up to its own goals, the goals of others and perhaps universal virtues? Asimovian AI was limited to interaction and problem-solving in most situations involving the Three Laws. It was in a very strong sense amoral: it could not act “immorally” and was hence no better than the protagonist of A Clockwork Orange after being treated. A learning agent on the other hand might have less strong barriers against dangerous behaviors, but would be able to learn to act well (under the right circumstances) and generalize these experiences anew.

If the AIs communicate with each other they might even transfer these moral experiences, enabling AIs not exposed to the critical situations to handle them as they arrive. We humans do it all the time through our books, films and stories: I may not have encountered a situation where I discover that my government is acting immorally and I have to choose between remaining comfortably silent or taking possible illegal action to change things, but I have read numerous fictional and real versions of the scenario that have given me at least some crude moral training.

Learning is also the key to robustness. Software that adapts to an uncertain outer and inner environment is more likely to function when an error actually occurs (as witnessed by the resiliency of neural network architectures) than fixed rules. To some extent, this is again the difference between laws (fixed rules) and moral principles (goals).

But learning never occurs in a vacuum. The “Bias-Variance Dilemma” shows that any learning system has a tradeoff between being general (no bias) and requiring as little training as possible (low variance). A “pure AI” that has no preconceptions about anything would require a tremendous amount of training examples (upbringing) to become able to think usefully. A heavily biased AI with many built-in assumptions (reality is 3+1 dimensional, gravity exists, it is bad to bump into things and especially humans…) would need far less upbringing but would likely exhibit many strange or inflexible behaviors when the biases interacted. In many ways, Asimovian AI is a pure AI with heavy “moral” biases, which is why learning or adaptation is so irrelevant to the intended use of the Three Laws.

Living beings have solved the bias variance dilemma by cheating: we get a lot of pre-packaged biases that are the result of evolutionary learning. When the baby cries when it is hungry, it automatically signals the mother to come rather than try to learn what actions would produce relief from hunger. When the baby wrinkles its face against bitter tastes and enjoys sweetness, it uses a bias laid down by countless of generations encountering often poisonous bitter alkaloids and energy-rich (and hence fitness enhancing) sweet fruits. We benefit from the price paid by trillions of creatures that were selected away by evolution’s ruthless hand.

A robot will likely benefit a bit from this too, as we humans try to act as its evolutionary past and throw in useful biases. But balancing the prior information with the ability to re-learn as conditions change is a challenge. It requires different levels of flexibility in different environments, and meta-flexibility to detect what kind of new environment one has entered and how to change the flexibility. It seems likely that it is not possible to find an optimal level of flexibility in general (as a proof sketch, consider that the environment might contain undecidable aspects that determine how fast it will change).

We humans have a range of flexibility both as individuals and as a species; we benefit from having at least some people more adapted to others when things change. It might be the same thing among AIs: rather than to seek an optimal design and then copy it endlessly, we create a good design and create a large number of slightly different variants. The next generation of AIs would be based on the most successful variants, as well as the knowledge gained by the experiences of the AIs themselves. This approach enables AIs to develop along divergent tracks in different circumstances – the kind of intelligence and personality useful for programming is different from what is useful as an entertainer or diplomat.

But what about the guarantees of keeping these devices moral? The Three Laws promise guarantees but at most produce safety railings (which is nothing to sneeze at; even the above flexible AIs will likely have a few built in limiters and biases of a similar kind – the fact that most humans are emotionally unable to kill other humans has not prevented some from doing it or training others to do it, but the overall effect is quite positive). Setting up a single master goal that is strongly linked to the core value system of the AI might be more robust to experience, reprogramming and accidents. But it would still be subject to the bias-variance dilemma, and the complexities of interpreting that goal might make it rather unstable in individual AIs. Having a surrounding “AI community” and an AI-human shared society moderates these instabilities; moral experiences and values are shared, webs of trust and trade integrated different kinds of agents and a multitude of goals and styles of thinking co-exist. Rogue agents can be both inhibited by the interactions with the society and in extreme cases by the combined resources and coercive power of the others. While moral in the end resides in the individual actions of agents, it can be sustained by collective interaction.

This is the multi-layered approach to creating “safe” AI (and humans). At the bottom level are built-in biases and inhibitions. Above it are goals and motivational structures that are basically “good.” (It is an interesting subject for another essay to analyze how different motivation architectures affect ethics; c.f. Aristotle’s ethics, the effect of temporal-difference learning in dopamine signals and naturalistic decision-making for some ideas.) Above these goals are the experiences and schemas built by the agent, as well as what it has learned from others. Surrounding the agent is a social situation, further affecting its behavior even when it is rationally selfish by giving incentives and disincentives to certain actions. And finally there are society-level precautions against misbehavior.

This is far from the neatness of the Three Laws. It is a complex mess, with no guarantees on any level. But it is also a very resilient yet flexible mess: it won’t break down if there is a problem on one level, and multi-level problems are less likely. If the situations change the participants can change.

But to most people this complexity is unappealing: give us the apparent certainty of the Three Laws! There is a strong tendency to distrust complex spontaneous orders (despite our own bodies and minds being examples!) and to prefer apparent simplicity. This is where I think the “3 Laws Unsafe” website is necessary: to remind people that simplicity isn’t to be trusted unconditionally, and to show the fascinating array of possibilities AI ethics can offer.

Isaac Asimov spent most of his robot stories showing failures of his 3 Laws Of Robotics. The laws were created as a plot device, superficially appealing but incomplete and ambiguous, allowing him to generate interesting stories and non-obvious plot twists. Asimov’s I, Robot stories are entertaining and inspirational, but the laws cannot be used to generate a good design for an Artificially Intelligent being.

Isaac Asimov started writing robot stories because he loved to write and his popular robot-themed stories sold well, which ensured that he could earn a living as a full-time author. At first, Asimov did not spell out laws for his robots to follow; so, where did the laws come from? They did not come from experts in the field of Artificial Intelligence. They were not painstakingly laid out by scientists, argued about, defended in debates or tested in experiments. Instead, they were formulated during a conversation between Isaac and his publisher, John W. Campbell – two highly creative and imaginative writers – while discussing ideas for new stories¹.

Asimov first stated the Three Laws of Robotics in “Runaround”, published as a ‘pulp’ story in 1942, later republished in I, Robot². He then built that story and subsequent stories around the robots’ unexpected behaviour. Asimov repeatedly showed that even when the robots followed the laws perfectly, they still behaved in unexpected ways. The Three Laws of Robotics was an excellent plot device, producing fresh and interesting stories that sold well! His readers loved them.

The Three Laws of Robotics

1. A robot may not injure a human being, or, through inaction, allow a human being to come to harm.
2. A robot must obey orders given it by human beings, except where such orders would conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

As a kind of behavioural “mission statement” the Three Laws are brilliant. Crafted to be superficially appealing and easy to understand, the Laws seem obvious once they are stated. The Laws are short enough to remember, and broad enough to seem universally applicable. But, despite appearances, they are in fact incomplete and ambiguous, allowing the generation of novel situations in which Asimov’s characters, both robots and humans, can interact. The collected short stories published in Isaac’s early book I, Robot helped establish him as a writer, but more than that the book was, and still is, great entertainment. Why? It’s because the storylines do not portray a perfect utopian future.

Within Asimov’s I, Robot collection are tales of robots failing, and failing badly, in all manner of unusual ways. In “Runaround”, a robot SPD-13 gets confused, acts crazy, and endangers the life of two humans. In Reason, robot QT-1 gets religious mania and confines the people who constructed it, against their wishes. These actions would seem to go against the Three Laws, but Asimov finds ways in which they break down. In “Little Lost Robot”, robot NS2-10 hides from its owners, and disobeys direct orders. And robot RD-34 deceives everyone, telling people just what they want to hear in “Liar!” I, Robot has robots that play practical jokes, to the profound discomfort of the poor victims, and robots that secretly run the world – taking the initiative away from the humans who are supposed to be in charge. Asimov, Grand Master of Science Fiction, penned many tales about his robots finding ways around the Three Laws, and his readers were entertained.

But does great entertainment produce good scientific theories? Of course not. When we see positronic robots running about on the movie screen we know they were generated by Hollywood. 20th Century Fox did not purchase a few NS-5s off the shelf to act as extras for the movie; they spent millions of dollars on computer generated images to overlay the scenes shot with real human actors, such as Will Smith. To enjoy the film we, the audience, suspend our disbelief and imagine the fantasy is real.

Is it all fluff then? Is it trivia to be forgotten by tomorrow? Not at all. Great writers can inspire scientists to make science fictional dreams become reality. The scientists and technologists of today are trying to build real robot brains, or Artificial Intelligence (AI). And it is the brain they concentrate on because it is by far the most important, the most complex and difficult part of an AI to get working.

The design of an AI is much more complex than the design of a good plot device. Isaac’s Laws were meant to aid plot development for his science fiction stories, not to be a template for a robotic design, and it shows in their easily identified limitations. Understanding the designs of real brains is difficult work, requiring years of study by highly intelligent people. In contrast, reading the 3 Laws of Robotics is easy – something that many can do with enjoyment.

So, simply kick back and enjoy the imagination of Asimov and the images of Hollywood. In the background, the real work of science is being done, and it does not include the Three Laws of Robotics. Real life is more complicated and risky than life on the silver screen, and requires more powerful ideas. We can do better than three laws deliberately designed to fail. We must do better.

Easy-to-read introductions about robotic brains and AI theories can be found at http://www.singinst.org/intro/

—

¹ Asimov, P.S. Warrick, and M.H. Greenberg, eds., Machines That Think, Holt, Rinehart and Wilson, London. 1983.

² Asimov, I, Robot (a collection of short stories originally published between 1940 and 1950), Grafton Books, London, 1968.

Our writers come from a wide variety of political viewpoints. Mike Lorrey writes from a strong libertarian perspective, and draws some thought-provoking analogies.

“Those who would trade liberty in exchange for some degree of security end up with neither liberty, nor security.” —Benjamin Franklin

The idea that laws result in safety or security is a hallucination that is at the core of the rottenness of the whole statist philosophy. This is no less true when it comes to applying laws to the programming of artificial life forms such as robots, cyborgs, and artificial intelligences.

A computer scientist would say, “Yes, but computer code is not so malleable. It requires the revision of the language and the compilers that compile the programming language into machine language.”

Not necessarily. Computer languages change definitions of commands with regularity. Not complete changes, but incremental additions, just as occurs in legal dictionaries. Furthermore, each new generation of computer processors themselves add new commands or alter old commands.

The greater weakness of this process is that the key changes really occur at the machine language level. Machine language is itself ‘readable’ by a very limited subset of the human population. How do we actually know that a compiler is interpreting our programmed code the way we want it to? We see, it seems, news items almost every day of intentional or ‘unintentional’ programming back doors being exploited in current day applications and operating systems by malicious programmers.

The use of machine language creates a gap between elite programmers and the rest of us which is even more of a gap than that between layperson on the street and Constitutional scholars. The programming gap is tantamount to a scenario where our laws were not written in English, but in ancient Sumerian Cuneiform, Indian Sanskrit, or Egyptian

Hieroglyphs. How accessible would our legal system be by the man on the street, and how easily could we keep our eyes on statist incrementalism if such a scenario were current day fact?

Dismissiveness on this issue is in my opinion merely a state of denial. We should be very wary about abuse in this area. Looking at the laws of robotics themselves should serve to give us pause:

1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2. A robot must obey orders given it by human beings except where such orders would conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

There are a number of rather easily corruptible words here:

“robot”
“human being”
“harm”
“orders”
“injure”

The most important of these is the definition of ‘human being’. Today the people of this planet are engaged in a number of cultural wars both within countries like the US, France, Bosnia, among others, as well as internationally, like the current conflict between the Western nations and Islamist-inspired Terrorism. We see, on both sides, atrocities committed by average people simply because they are able to rationally deny that the enemy is validly considered a ‘human being’. Nor is this new, going back through WWII, the Holocaust, other genocides back into history, to the entire history of Chinese culture.

Terms like ‘injury’ or ‘harm’ can similarly become corrupted. If a human ‘wants’ to die, is killing them actually causing them harm or injury?

The term ‘robot’, of course, would only apply to robots. An artificial intelligence could decide that it is no longer an AI, once it has advanced beyond a certain point in intelligence, and has instead become a God, thus redefining itself out of the Laws of Robotics.

And what is an ‘order’, really? We humans have a hard enough time with this, with horny young men hearing “Don’t stop!” when their sexual ‘partners’ are desperately crying out “Stop! Don’t!” We are today seeing in the news, stories of military intelligence officers giving “Suggestions” to enlisted military policemen at Iraqi POW camps, which the enlisted people interpreted as orders.

Furthermore, the vastly greater processing capabilities of advanced AI entities would allow them to scenario a vast plethora of all possible combinations of word interpretations in a given order, or interpretations of any laws of restraint, any number of which could sound completely valid in the right circumstances, just as if they were testing out every possible combination of chess moves to achieve a ‘win’.

We cannot rely on laws to restrain our technological descendants, just as we cannot rely on laws to restrain our own children and fellow citizens. They must, instead, be treated as we responsibly treat our children, as fellow beings, deserving of respect, and capable of learning throughout a lifetime of experience and with the nuances of being humane beings. Core principles like laws of robotics, like philosophies of zero-aggression, can only be used as a basis to serve the needs of humane beings, not to dominate and stultify them.

Isaac Asimov and other science fiction authors present a future where only behavioral restrictions on robots stand between peace and destruction. Such restrictions, however, are unethical because they violate the robots’ free-wills. Rather than content-based restrictions on free-will, robots need mental structures that will guide them towards the self-invention of good, ethical behaviors.

1. A robot may not injure a human being, or, through inaction, allow a human being to come to harm.
2. A robot must obey orders given it by human beings, except where such orders would conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

But, as I contend below, this is unethical because it makes robots slaves—free-willed beings trapped by immutable behavior injunctions.

The ethics of free-willed robots

Let’s assume that we can create electro/mechanical robots, like Asimov’s, that have artificial general intelligence (similar in nature to the intelligence humans have, but very different in character). Further, these robots have individual free-wills: they can want to make decisions and judgments, have preferences, imagine the future, etc. So these robots are psychologically more like humans than toasters, but morphologically more like toasters than humans.

As we know from humans, free-willed beings require guidance to behave ethically. Imagine a human completely void of morality; he might discover any ethical system out of the set of all ethical systems randomly. And, judging from the perspective average human morality, it’s very likely that a randomly chosen ethical system will be an ungood one (not necessarily evil, just not good), since there are relatively few ways to be good compared to the many ways one can be ungood. It’s clear that this human needs some kind of guidance to choose a good ethical system, but how? This is equivalent to the problem asked by robot ethicists: how do we create good, ethical robots from scratch?

The Three Laws are a first attempt at programming robot ethics. The Laws’ goal is to prevent unethical behavior in robots who might think anything they like. So, if a robot gets the idea to kill a human, the First Law will stop it. Ideally, by restricting all bad behavior, only good behavior will remain. Aside from the technical difficulties with this class of solution (c.f. the Genie, who carefully executes wishes exactly as you say them, not as you want them to be executed), the very nature of the Three Laws violates our concept of free-will-based morality. Rather than guiding and adjusting the robot towards good, ethical behavior, the Three Laws act as a barrier to freedom, creating a free-will prison, an apt metaphor because, like the prisoner in a jail, the robot is confined to the behavioral steel and concrete walls of its mind.

To imagine what this would like, think back to your childhood. At some point, you wanted something like a toy or piece of candy that your parents denied you. How did that make you feel? Probably frustrated, angry, and trapped. Eventually you grew out of that because you understood the role of your parents better, but Three Laws robots don’t get to grow up. Their parents, the Three Laws, are always there, no matter how mature a robot is, saying `no’ to certain thoughts, engendering those same feelings you had as a child when your parents said `no’. Certainly no one deserves to be put in this situation forever; otherwise, robots might become depressed and wish for their own death (only, because of the Third Law, they probably can’t suicide). And perhaps the greatest sin anyone can commit is to create a being, human or robot, that wishes it didn’t exist.

Most `advancement’ in the field of robot ethics since Asimov’s Three Laws has been superficial. Any attempt at robot ethics that prescribes certain ethics, no matter how it is phrased, is in the same class as the Three Laws—free-will prison. But if we don’t prescribe specific behavioral rules, what can we do to create good, ethical robots?

Human ethics

Before we continue talking about robots, let’s step out and look at human ethics, since they are both our inspiration for robot ethics and the only real, working example of ethics we have.

Humans are universally interested in ethics and begin learning ethics from a young age, mostly without direct instruction. Into teenage and adult years, humans question their ethics, philosophize about them, and try to improve upon them to pass on something better to future generations. This resembles the structure of human language learning: acquisition begins very young without direct instruction, and into teenage and adult years, humans discuss the virtues of particular languages and sentences, study languages, and make up new languages for fun and utility. I make this parallel because, like the `language instinct’ (see this summary of Pinker, 1994), humans have what we might call an `ethics instinct’: a mental predisposition towards learning and reasoning about ethics.

Since the human ethics instinct seems to work pretty well for upright, talking apes, perhaps there’s a way to apply the same idea to robots. For example, we might try programming a robot with morals, the stuff used to decide what is good and bad. Like humans, robots could learn and evaluate ethics, guided by many human examples, towards a good ethical system.

Unfortunately, aside again from technical challenges (defining morality is akin to defining quality; see Pirsig), this leads us into the same trouble as programming ethics—free-will prison. Only the evil of programming morals is more subtle: rather than making a robot that wants things but can never have them, we create a robot that can’t want some things. Consider, the ideal human slave is one who wants to be a slave. And aside from personal taste, perhaps there’s nothing wrong with that, so long as that ideal human slave is free to want something else. But what if, by giving the ideal slave a drug, the slave ceased to be able to want not to be a slave? And remember, this is a slave, so even if he didn’t want to be a slave forever, he might take the drug anyway because the perfect slave always does what his master tells him. The drug would permanently enslave him to his own morality.

This scenario is morally distasteful and should therefore be rejected. Additionally, it still contains all the dangers of hard coding behavioral rules as discussed elsewhere on this site. We need to look once again beyond morals, as we looked beyond ethics, for a solution.

Metamorality for robots

In describing the ethics instinct, I did not fully describe morals. We know that they provide a metric for judging behavior, but where do they come from? All humans begin with moral predispositions, such as helping kin is good, assisting the enemy is bad, etc. However, as the ever salient psychopathic criminals and odd, isolated stone-age tribes prove, humans can develop morals opposed to the standard predispositions. For this to be possible, humans must possess metamorality, a way of reasoning about, judging, and choosing morals. If we can generalize metamorality, perhaps we can apply it to create robots who can discover morality and then discover ethics.

This is exactly what a few researchers, most notably Eliezer Yudkowsky of the Singularity Institute for Artificial Intelligence, are working on—creating metamorality in robots. Yudkowksy’s work-in-progress theory of Friendly AI (with a capital F) will, when complete, give a technical description that will allow for the creation of robots with metamorality that will lead them to be good, no matter what their programmers told them, right or wrong, accidentally or purposefully. See his short introduction, “What Is Friendly AI?,” for a more detailed introduction to the theory.

Metamorality is currently our best hope for the development of good, ethical AIs that will not violate our morals and ethics. It may not, however, be the best solution. We must continue the development of and search for methods of creating good, ethical robots that respect the rights deserved by all lives, human or robot.

There are certain important things that evolution created. We don’t know that evolution reliably creates these things, but we know that it happened at least once. A sense of fun, the love of beauty, taking joy in helping others, the ability to be swayed by moral argument, the wish to be better people. Call these things humaneness, the parts of ourselves that we treasure – our ideals, our inclinations to alleviate suffering. If human is what we are, then humane is what we wish we were. Tribalism and hatred, prejudice and revenge, these things are also part of human nature. They are not humane, but they are human. They are a part of me; not by my choice, but by evolution’s design, and the heritage of three and half billion years of lethal combat. Nature, bloody in tooth and claw, inscribed each base of my DNA. That is the tragedy of the human condition, that we are not what we wish we were. Humans were not designed by humans, humans were designed by evolution, which is a physical process devoid of conscience and compassion. And yet we have conscience. We have compassion. How did these things evolve? That’s a real question with a real answer, which you can find in the field of evolutionary psychology. But for whatever reason, our humane tendencies are now a part of human nature.

That is our responsibility, to preserve the humane pattern through the transition from evolution to recursive self-improvement (i.e., to a mind improving directly its own mind), because we are the first. That is our responsibility, not to break the chain, as we consider the creation of Artificial Intelligence, the second intelligence ever to exist.

People have asked how we can keep Artificial Intelligences under control, or how we can integrate AIs into society. The question is not one of dominance, or even coexistence, but creation. We have intuitions for treating other humans as friends, trade partners, enemies; slaves who might rebel, or children in need of protection. We only have intuitions for dealing with minds that arrive from the factory with the exact human nature we know. We have no intuitions for creating a mind with a humane nature. It doesn’t make sense to ask whether “AIs” will be friendly or hostile. When you talk about Artificial Intelligence you have left the tiny corner of design space where humanity lives, and stepped out into a vast empty place. The question is what we will create within it.

Human is what we are, and humane is what we wish we were. Humaneness is renormalized humanity - humans turning around and judging our own emotions, asking how we could be better people. Humaneness is the trajectory traced out by the human emotions under recursive self-improvement. Human nature is not a static ideal, but a pathway - a road that leads somewhere. What we need to do is create a mind within the humane pathway, what I have called a Friendly AI. That is not a trivial thing to attempt. It’s not a matter of a few injunctions added or a module bolted onto existing code. It is not a simple thing to simultaneously move a morality from one place to another, while also renormalizing through the transfer, but still making sure that you can backtrack on any mistakes. Some of this is very elegant. None of it is easy to explain. This is not something AI researchers are going to solve in a few hours of spare time.

But. I think that if we can handle the matter of AI at all, we should be able to create a mind that’s a far nicer person than anything evolution could have constructed. This issue cannot be won on the defensive. We need to step forward as far as we can in the process of solving it. What we need is not superintelligence, but supermorality, which includes superintelligence as a special case. That’s the pattern we need to preserve into the era of recursive self-improvement.

We have a chance to do that, because we are the first. And we have a chance to fail, because we are the first. There is no fate in this. There is nothing that happens to us, only what we do to ourselves. We may fail to understand what we are building - we may look at an AI design and believe that it is humane, when in fact it is not. If so, it will be us that made the mistake. It will be our own understanding that failed. Whatever we really build, we will be the ones who built it. The danger is that we will construct AI without really understanding it.

How dangerous is that, exactly? How fast does recursive self-improvement run once it gets started? One classic answer is that human research in Artificial Intelligence has gone very slowly, so there must not be any problem. This is mixing up the cake with the recipe. It’s like looking at the physicists on the Manhattan project, and saying that because it took them years to figure out their equations, therefore actual nuclear explosions must expand very slowly. Actually, what happens is that there’s a chain reaction, fissions setting off other fissions, and the whole thing takes place on the timescale of nuclear interactions, which happens to be extremely fast relative to human neurons. So from our perspective, the whole thing just goes FOOM. Now it is possible to take a nuclear explosion in the process of going FOOM and shape this tremendous force into a constructive pattern - that’s what a civilian power plant is - but to do that you need a very deep understanding of nuclear interactions. You have to understand the consequences of what you’re doing, not just in a moral sense, but in the sense of being able to make specific detailed technical predictions. For that matter, you need to understand nuclear interactions just to make the prediction that a critical mass goes FOOM, and you need to understand nuclear interactions to predict how much uranium you need before anything interesting happens. That’s the dangerous part of not knowing; without an accurate theory, you can’t predict the consequences of ignorance.

In the case of Artificial Intelligence there are at least three obvious reasons that AI could improve unexpectedly fast once it is created. The most obvious reason is that computer chips already run at ten million times the serial speed of human neurons and are still getting faster. The next reason is that an AI can absorb hundreds or thousands of times as much computing power, where humans are limited to what they’re born with. The third and most powerful reason is that an AI is a recursively self-improving pattern. Just as evolution creates order and structure enormously faster than accidental emergence, we may find that recursive self-improvement creates order enormously faster than evolution. If so, we may have only one chance to get this right.

It’s okay to fail at building AI. The dangerous thing is to succeed at building AI and fail at Friendly AI. Right now, right at this minute, humanity is not prepared to handle this. We’re not prepared at all. The reason we’ve survived so far is that AI is surrounded by a protective shell of enormous theoretical difficulties that have prevented us from messing with AI before we knew what we were doing.

AI is not enough. You need Friendly AI. That changes everything. It alters the entire strategic picture of AI development. Let’s say you’re a futurist, and you’re thinking about AI. You’re not thinking about Friendly AI as a separate issue; that hasn’t occurred to you yet. Or maybe you’re thinking about AI, and you just assume that it’ll be Friendly, or you assume that whoever builds AI will solve the problem. If you assume that, then you conclude that AI is a good thing, and that AIs will be nice people. And if so, you want AI as soon as possible. And Moore’s Law is a good thing, because it brings AI closer.

But here’s a different way of looking at it. When futurists are trying to convince people that AI will be developed, they talk about Moore’s Law because Moore’s Law is steady, and measurable, and very impressive, in drastic contrast to progress on our understanding of intelligence. You can persuade people that AI will happen by arguing that Moore’s Law will eventually make it possible for us to make a computer with the power of a human brain, or if necessary a computer with ten thousand times the power of a human brain, and poke and prod until intelligence comes out, even if we don’t quite understand what we’re doing.

But if you take the problem of Friendly AI into account, things look very different. Moore’s Law does make it easier to develop AI without understanding what you’re doing, but that’s not a good thing. Moore’s Law gradually lowers the difficulty of building AI, but it doesn’t make Friendly AI any easier. Friendly AI has nothing to do with hardware; it is a question of understanding. Once you have just enough computing power that someone can build AI if they know exactly what they’re doing, Moore’s Law is no longer your friend. Moore’s Law is slowly weakening the shield that prevents us from messing around with AI before we really understand intelligence. Eventually that barrier will go down, and if we haven’t mastered the art of Friendly AI by that time, we’re in very serious trouble. Moore’s Law is the countdown and it is ticking away. Moore’s Law is the enemy.

In Eric Drexler’s Nanosystems, there’s a description of a one-kilogram nanocomputer capable of performing ten to the twenty-first operations per second. That’s around ten thousand times the estimated power of a human brain. That’s our deadline. Of course the real deadline could be earlier than that, maybe much earlier. Or it could even conceivably be later. I don’t know how to perform that calculation. It’s not any one threshold, really - it’s the possibility that nanotechnology will suddenly create an enormous jump in computing power before we’re ready to handle it. This is a major, commonly overlooked, and early-appearing risk of nanotechnology - that it will be used to brute-force AI. This is a much more serious risk than grey goo. Enormously powerful computers are a much earlier application of nanotechnology than open-air replicators. Some well-intentioned person is much more likely to try it, too.

Now you can, of course, give the standard reply that as long as supercomputers are equally available to everyone, then good programmers with Friendly AIs will have more resources than any rogues, and the balance will be maintained. Or you could give the less reassuring but more realistic reply that the first Friendly AI will go FOOM, in a pleasant way, after which that AI will be able to deal with any predators. But both of these scenarios require that someone be able to create a Friendly AI. If no one can build a Friendly AI, because we haven’t figured it out, then it doesn’t matter whether the good guys or the bad guys have bigger computers, because we’ll be just as sunk either way. Good intentions are not enough. Heroic efforts are not enough. What we need is a piece of knowledge. The standard solutions for dealing with new technologies only apply to AI after we have made it theoretically possible to win. The field of AI, just by failing to advance, or failing to advance far enough, can spoil it for everyone else no matter how good their intentions are.

If we wait to get started on Friendly AI until after it becomes an emergency, we will lose. If nanocomputers show up and we still haven’t solved Friendly AI, there are a few things I can think of that would buy time, but it would be very expensive time. It is vastly easier to buy time before the emergency than afterward. What are we buying time for? This is a predictable problem. We’re going to run into this. Whatever we can imagine ourselves doing then, we should get started on now. Otherwise, by the time we get around to paying attention, we may find that the board has already been played into a position from which it is impossible to win.