What Is the Intelligence Explosion? The Cambridge Paper and Its Critics

AI systems now write most of the code inside the companies that build them, and a September 28 paper by 22 researchers asks what happens if they take over the research too. The document says an intelligence explosion could follow, with conditions attached; this week's skeptics say the loop it depends on has not shown up in the data.

The idea is from 1965; the evidence is from this year

The statistician I.J. Good wrote the passage that started this vocabulary in 1965:

“Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion', and the intelligence of man would be left far behind.”

The mechanism in that sentence is a feedback loop. A machine that is good at designing machines designs a better one, and the better one designs a better one still, with each generation shortening the time to the next. Good thought the first such machine would be “the last invention that man need ever make.”

For decades the argument stayed where Good left it: a chain of logic with nothing to measure. That changed this year. The paper, published on September 28 by the Cambridge Programme on AI Science and Policy with 22 authors, opens with a claim about the present, not the future: “In contrast to even a year ago, AI systems now write most of the code inside the companies that build them.”

What changed in 2026 is not the argument; it is that the automation the argument depends on can now be measured.

What the paper actually claims

The paper's title asks a question: “What if automating AI R&D triggers an intelligence explosion?” Its answer has two parts, and headlines tend to keep one of them.

The first part is a forecast about work, not about intelligence. The authors write that “AI systems are on track to automate most AI R&D work within a few years, and possibly all of it.” That is a statement about who does the next round of AI research, and it is the part they treat as supported.

The second part is the explosion itself, defined as a situation where “years of advances are compressed into months or less.” The paper attaches it to the first part with conditionals: if the automation triggers it, benefits could arrive sooner, and three risks could sharpen. Capabilities could grow faster than society can keep up with. People could lose control of systems more capable than themselves. And the usual checks on power, between countries, companies and branches of government, could erode.

The authors also narrow the mechanism. The paper focuses on a software-driven explosion, where research automating research speeds up progress through better methods alone, rather than through more hardware. And it carries its own caution label: “Although there remains much uncertainty about these possibilities, the high stakes warrant serious further attention.”

The paper forecasts the automation of research work and treats the explosion as one possible consequence of it.

The skeptics: diminishing returns and a bank's base case

The pushback arrived quickly, and it draws on the same kind of lab data the paper cites.

Ramez Naam, an investor and computer scientist, published a skeptical essay, “Where's the 'intelligence explosion'?”, the day before the Cambridge paper appeared. His argument: look at the performance data the labs publish in their own system cards, and the gains show steep diminishing returns. In his reading, the feedback loops an explosion requires are too weak in current systems to sustain runaway growth. A video version of his case ran on October 10.

Deutsche Bank's research unit reached a neighboring conclusion from an economics angle. A report by Adrian Cox and Azeem Azhar names recursive self-improvement, “an AI tool is able to continuously and autonomously design and improve itself,” as the mechanism that would matter, and puts it in doubt: their base case is that “due to fundamental limitations on the speed AI programs and systems can achieve, we won't reach superintelligence.” The bank's Jim Reid adds a jobs note: across 250 to 300 years of new technologies, productivity gains kept creating demand for different work. “We all collectively lack the imagination to wonder what the jobs of the future will be,” he said.

Nvidia's CEO Jensen Huang has been the loudest industry skeptic. He said on a podcast in September: “I hate his predictions … All of his predictions have been wrong,” referring to Hinton, whose risk estimates Huang called not grounded in science.

The skeptics' doubts concentrate on one link: whether improvement really feeds on improvement at an increasing rate.

What the paper asks governments to do

The paper ends in policy, and the asks are specific. Policymakers, it says, should get visibility into how much AI research is automated, develop ways to steer and constrain an explosion if one starts, and prepare society for the impacts. No one is asked to stop building AI.

The ask lands in a month when the machinery already exists: the US federal government mandated the phrase “Super Intelligence” in federal documents in September, a task force called the Super Intelligence Force has been announced with a report due, and a proposed federal definition of the term is expected around November 28. (We explain the order in our EO post, the task force in our SIF explainer, and the definition fight in our definition post.)

Hinton, the paper's most famous co-author, keeps his own version of the argument in public. This week he told the Guardian: “We're creating alien beings,” and “we only have a couple of years to adapt to that.” He is precise about what such numbers mean: “I'm trying to convey that there is a really serious possibility of something happening. If, for example, you're told there was a 10% possibility of a bus crashing, you wouldn't get on the bus.”

The paper's authors are not asking governments to stop AI research; they are asking for instruments: visibility, steering, and preparation.

FAQ

Sources

1. I.J. Good, “Speculations Concerning the First Ultraintelligent Machine,” 1965; see the “Intelligence explosion” section of Wikipedia's singularity article. en.wikipedia.org
2. Chan, Winter, Barto, Pachocki, Hinton, Horvitz, Bengio, Song, Clark, et al., “What if automating AI R&D triggers an intelligence explosion?” arXiv, submitted Sept 28, 2026. arxiv.org
3. The same paper on the CASP site, with the WSJ and Guardian coverage listed. casp.ac
4. The Guardian, “AI godfathers warn of runaway 'intelligence explosion',” Sept 28, 2026. theguardian.com
5. The Guardian, “AI godfather Geoffrey Hinton: 'We're creating alien beings...'”, Oct 13, 2026. theguardian.com
6. MarketWatch (Dow Jones), “'Superintelligence' may never arrive, Deutsche Bank says,” Oct 13, 2026. marketwatch.com
7. Ramez Naam, “Where's the 'intelligence explosion'?”, Noahpinion, Sept 27, 2026. noahpinion.blog
8. “The Lab Data Proves the Intelligence Explosion Is a Myth,” Ramez Naam video, Oct 10, 2026. youtube.com