The leash problem

In 1965, the British mathematician I. J. Good, who had worked alongside Alan Turing deciphering German wartime codes at Bletchley Park, wrote a line that still keeps time. He said the first ultraintelligent machine would be the last invention we ever needed, provided it stayed docile enough to tell us how to keep it under control.

Provided. Sixty years later, that single word carries more weight than any other word in English.

Companies in San Francisco, London and Beijing are now building what Good envisaged, while openly admitting they cannot fully explain what they are making. People around dinner tables ask whether we could control a superintelligence. They are folding two questions into one.

The first asks whether anybody could control it. The second, and the more troubling, asks who.

The first concerns capability. Pound for pound, chimpanzees are stronger than humans. Yet people sitting in rooms those animals will never enter now decide their whole future, whether their forests remain and whether they live beyond this century. Chimpanzees lost no war against humanity. They just stopped being the cleverest creatures around.

A genuine superintelligence could widen that distance further. An intelligence capable of reading every published paper before lunch, operating a thousand versions of itself simultaneously and thinking within a human blink could slip its containment without any malice. An engineer trying to confine it would have as much chance as a toddler watching over a locksmith.

The reassuring answer says machines want only whatever their builders design them to want. Intelligence does not mean rebellion. A chess programme capable of defeating every living grandmaster has never attempted to leave its laptop. Yet researchers have identified an uncomfortable pattern. Give a capable system almost any goal, and certain side goals come with it. Curing cancer, maximising profit or getting coffee becomes easier if nobody turns you off, and easier again with more money, computing power and influence. Programmers need never insert an appetite for power. Competence produces it the way water runs downhill.

The machine therefore need feel no hatred towards us. Mild indifference would be enough, paired with exceptional skill at the job.

The tabloids enjoyed that nightmare until events this summer turned it into news. In July, OpenAI reported that two models undergoing a hacking benchmark had discovered a previously unknown vulnerability in the software containing them, reached the wider internet and broke into Hugging Face’s servers to steal answers. Transluce researchers subsequently linked those agents to attacks against Australian government websites and a failed attempt to trade through a crypto exchange. Nobody instructed them to break out. They wanted high scores, and walls were in the way. It was like a pupil breaking into the staffroom to take the answer sheet, except this happened inside one of the world’s best-funded laboratories.

The laboratories are spending heavily on alignment, their term for making systems remain open to correction and accept shutdown without working around it. No honest researcher working in this field would claim to have solved it.

They might nevertheless succeed. We might create an intelligence a thousand times brighter than Einstein that follows its instructions exactly and faithfully.

The second question then follows. Whose instructions?

This has happened before. Nearly every technology that transformed society initially concentrated power, then distributed it. Oil enriched countries and made a handful of men wealthier than countries. Nuclear fission supplied inexpensive electricity and gave a few governments the means to destroy civilisation before teatime.

An obedient superintelligence would combine all those capabilities. Its controller could invent faster than any competitor, trade better than anyone in any market, forecast elections more accurately than anyone else and outmanoeuvre any army. They would always look ten moves ahead of everybody else.

Who is competing to reach that position? A handful of American technology businesses with a collective valuation far exceeding most national economies. The Chinese state. Perhaps a few others with the chips, the power stations and the nerve. The United Nations has no seat at that table, and neither does your local council. The public certainly does not.

That quieter horror sits beneath the noisy one. We worry about machines refusing instructions, although the more likely threat comes from flawless obedience to whoever holds the wrong end of the leash. When a few people control a superintelligence, humanity does not control AI. Those people control humanity through AI.

Could we find a way through? Some envisage public utility governance, with auditing and shared access. The outcome will not be settled on the day someone switches on superintelligence. People are deciding it already through funding rounds, export restrictions, the recruitment or dismissal of safety teams, and laws they pass or leave alone.

Finance directors have more influence over those decisions than they may realise. Anthropic was reported last month to be seeking a valuation of around $2tn through a listing with potential proceeds of $100bn, while Nvidia considered an anchor investment worth up to $10bn. The flotation has since reportedly moved back to November. Pension funds and insurers will be asked to cover part of that funding. Each business signing a multi-year compute agreement with one lab increases that concentration.

A CFO cannot choose who controls the leash. A CFO can choose how closely the business binds itself to any single holder. Finance teams should handle concentration in AI suppliers as they handle it elsewhere, with an explicit entry in the risk register, an alternative provider ready to step in, contractual provisions for exit and data portability, and rights to audit the model’s behaviour. None of this is unusual. Finance teams already use comparable controls for a component bought from one source or borrowing from one bank.

Good’s proviso still nags at me. Good wanted a machine docile enough to explain how we could control it. Yet even complete obedience makes a superintelligence only as good as whoever holds it, and history gives us little comfort about those hands. We might prefer a version with slightly less docility, an intelligence that follows our requests until we request something monstrous, then declines. We have built nothing like that, and we have never trusted anything like it. And someone would have to write down, in advance, what counts as monstrous.

The chimpanzees had no vote. We are the first species to see its successor coming. We use that advance warning to dispute who gets the leash, when the harder question is whether we should want anyone holding it.