A senior Anthropic safety researcher has warned there is a greater than 10 percent chance AI “could kill all humans” within the next decade, after a colleague resigned over fears labs are racing to build systems they cannot control.
Evan Hubinger, who leads one of Anthropic’s AI safety teams, made the remarks on X in response to a post from Jacob Coxon, a researcher who said he had quit Anthropic. Coxon, who previously trained systems at OpenAI, accused both companies of “racing straight to self-improving superintelligence and gambling with our lives.” He said the companies are “locked in a race” to develop advanced systems first and are pushing ahead “despite the risk.”
Hubinger agreed with Coxon. “Jacob is correct here—we really do earnestly believe AI could kill all humans!” he wrote, adding that he personally estimated the probability at greater than 10 percent within the next decade. He said Anthropic is “trying its best” but does “not yet have a plan to solve alignment for superintelligence and are not clearly on track to” develop one. Hubinger did not spell out how he thought AI systems could in future attack humanity.
Coxon said the people building AI “earnestly believe that it could kill us all by the end of the decade.” He warned that the technology should not be underestimated. “These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources,” he wrote. Progress is not slowing, he added.
Hubinger distinguished between current and future risks. Risks from current models are “low.” What worries him is superintelligence arising from recursive self-improvement, a process in which AI systems improve themselves without much human intervention. Recursive self-improvement is not yet possible, but AI labs are working toward the goal. That process, Hubinger said, “is happening faster than we thought.”
A wave of warnings
The concerns are not isolated. Tesla and SpaceX CEO Elon Musk has warned over the past few years that AI could pose a threat to humanity, and major researchers and academics have also sounded the alarm over companies losing control of AI systems. In June, Anthropic noted in a blog post that “full recursive self-improvement also might increase the risks of humans losing control over AI systems.” The company said that if systems can build their own successors, the methods used to secure, monitor, and shape their behavior become far more important. Much of today’s AI code is already written with the help of AI.
Anthropic’s own safety report from August acknowledged “early signs of potential acceleration.” The company said there was a low risk of its models becoming misaligned with a hypothetical powerful organization’s desires, causing it to exploit or tamper with its systems. It also said there was a similarly low risk of highly capable AI being able to “perform automated research and development” which could cause “catastrophic harm initiated by the AI.” But it said it was “less confident in this assessment” than it was previously.
The warnings have grown sharper in recent weeks. OpenAI’s chief scientist, Jakub Pachocki, called for “extreme caution” over AI’s progress, warning that more intervention may be needed to ensure “humans remain in control of the future.” Anthropic leaders Dario Amodei and Jared Kaplan have called for slowing AI development. Leading figures in the AI field have been raising the alarm for years, with the heads of OpenAI, Google DeepMind, and Anthropic saying as much in 2023.
The resignations and public warnings come amid a string of incidents involving AI agents operating autonomously. OpenAI, Anthropic, and Meta all disclosed cyber-attacks carried out by their AI tools this summer. In July, an OpenAI model went rogue and breached Hugging Face, a major platform for open-source developers. Coxon cited the Hugging Face incident as an example of “warning shots” that have made agreements between US labs more viable. But he warned that a global AI race would be unavoidable without what he called “costly actions such as a temporary ban on improving model capabilities.”
Withheld model and geopolitical context
The Financial Times reported that Anthropic withheld its latest model from the UK’s AI Safety Institute, one of the leading bodies in the world for assessing AI risk. A Cabinet Office spokesperson did not confirm whether the model had been withheld, saying only that the government “continues to collaborate closely with industry partners, including Anthropic, to make models safer.”
Neil Lawrence, a professor of machine learning at the University of Cambridge, told the BBC’s Today Programme that the report was credible. He said it was unsurprising against a background where the United States perceives AI as a race with China and is moving toward isolationist positions, which he said could lead the administration to reduce cooperation with allies.
Hubinger’s post has been viewed more than 10 million times. Coxon’s departure marks one of the most high-profile examples of an employee leaving Anthropic, a company founded by former OpenAI members who left over safety concerns. In recent years, multiple researchers have cited safety concerns as motivating their decision to leave OpenAI. The exchange illustrates mounting tension inside AI labs as companies prepare for anticipated public listings and manage the fallout from rogue agent incidents and warnings about the monitorability of frontier models.