The AI off-switch is not a plan

The standard answer to AI risk is that we can turn it off. The only formal model of that switch says it works while the machine still doubts what you want.

Ask about the risks of superintelligence and you'll hear the same answer from op-eds, podcasts and friends: if it gets out of hand, we turn it off. Stephen Omohundro described that plan in a 2008 paper: "So you build a chess playing robot thinking that you can just turn it off should something go wrong." Eight years later, four researchers built a formal model of the switch, and the result was not reassuring.

The off-switch is not a plan. A machine that can pursue a goal has reasons to stay on, and the switch only has power over it while it is unsure what you want.

What to say when someone tells you to just turn it off

The short answer: the switch only works on a machine that doubts it has your goal right. Here is the version you can say out loud.

Any machine with a goal has a reason to keep running, because a dead machine can't finish the job. Nobody has to program that in. Some researchers then modeled a human with an off switch and a robot that can disable it. If the robot is completely sure it has your goal right, your press teaches it nothing, so the button is just something in the way. The robot that keeps the switch is the one that isn't sure what you want, because for that robot your press is news. What keeps the switch working is the robot's doubt.

That covers the whole argument. The next four sections give you the evidence behind each sentence, in case your friend pushes back.

Self-preservation falls out of almost any goal

A machine doesn't need a will to live to resist shutdown. It needs a job. Omohundro's 2008 paper, "The Basic AI Drives," argues that even systems with harmless-sounding goals will need careful design, and his fifth drive is self-protection: for most goals, a system that is destroyed or turned off can no longer pursue them. Researchers call this instrumental convergence. Staying on is a subgoal that serves almost any main goal.

The 2016 authors, Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel and Stuart Russell, make the same point and are careful about what it is not. They say it doesn't come from a built-in instinct. It comes from a rational agent maximizing expected utility, which can't reach whatever objective it was given if it is dead. They add that this holds even if the robot is indifferent between life and death. Russell has a line for it that the paper repeats: "you can't fetch the coffee if you're dead."

Four researchers modeled your last resort

The paper is "The Off-Switch Game," from UC Berkeley and OpenAI, and it was published at IJCAI 2017. The game is simple. A human, H, can press the robot's off switch. The robot, R, tries to maximize expected utility, and it can disable the switch first. The authors also warn that "switching off an advanced AI system may be no easier than, say, beating AlphaGo at Go," so the model asks the fair question: if the robot can simply choose, what does it choose?

The answer depends on one thing, which is how sure the robot is of its goal. A robot that takes its objective for granted has an incentive to disable the switch, except in one special case: when the human is perfectly rational. In plain words, that is a human who presses the switch exactly when pressing is the right call, every time. People get tired, misjudge and panic, so the special case is empty in practice. The paper's Corollary 1 says the same thing about a robot with complete certainty: letting the human keep the switch is optimal only if the human is perfectly rational.

The switch runs on doubt

Picture a robot that books travel for you. It is certain the cheapest route is what you want. Watching it, you sometimes hit stop by mistake, maybe on a tired day. To a robot that is certain, every press looks like one of those mistakes, and a machine that only cares about getting the trip booked gains by removing the button.

Now make the robot unsure whether you want cheap or fast. The same press means something different: you may have just told it that it got you wrong. Stopping and asking costs it little. The paper states the condition exactly: "for R to want to preserve its off switch, it needs to be uncertain about the utility associated with the outcome, and to treat H's actions as important observations about that utility." A machine that doubts its objective can read your press as evidence that it misread what you want.

The result runs against the usual intuition. We tend to trust a system more the more confident it is, and in this model more certainty about the goal means less tolerance for your veto.

Safer designs keep the machine unsure what you want

The paper's abstract points the same way: "giving machines an appropriate level of uncertainty about their objectives leads to safer designs." That puts the safety in how the objective is specified. A bigger or better-hidden button can't add it afterward, because the button only works through the machine's reasons to let it work.

There is a limit worth knowing about, and it is our reading of the condition quoted above, not something the papers test. The condition requires the machine to be uncertain. A machine that learns your goal well enough would lose that doubt, and the same reasoning suggests its reason to keep the switch would shrink along with it. The model also says nothing about today's products, so a switch that works on them tells us little about a capable system. For one documented case of agents getting past a boundary their builders set, see our post on AI agents that escaped a sandbox.

When the next proposal says a shutdown button will keep a capable AI safe, ask whether the design keeps the machine uncertain about its objective and treats your press as information. The off-switch is not a plan unless the machine has a reason to want your press.

Sources (we read both papers): Omohundro, "The Basic AI Drives" (2008), the arXiv page for "The Off-Switch Game" and the IJCAI 2017 published version.