Anthropic CEO Dario Amodei published an essay on Saturday calling for the AI industry to slow the pace of model development and submit to independent safety evaluations, a proposal that drew support from rival executives and sharp criticism from observers who questioned its motives.
The essay, titled “We Must Pace the Frontier,” laid out a three-part plan. Amodei said Anthropic would “unilaterally” commit to the first step: providing third-party evaluators, including organizations like METR, with permanent, employee-level access to its systems so they can verify adherence to safety measures, report on incidents, and assess models’ alignment during training. He compared these embedded evaluators to regulators placed inside banks, saying they would receive company badges, desks, and laptops with access mostly comparable to internal risk assessment teams, TechCrunch reported.
The second step calls for industry-wide coordination among AI companies in democratic countries to establish common safety standards and limits on unchecked progress. The third and most difficult step would seek global coordination, including cooperation with authoritarian governments like China and Russia, though Amodei acknowledged stark limits on what could be achieved.
Amodei’s appeal came amid a week of dire warnings. Jacob Coxon, an AI researcher who left Anthropic, told the BBC that “if we don’t slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future.” Coxon said people working at AI companies were “genuinely frightened” and concerned about the fate of humanity within the next two years. In posts reported by the Guardian, Coxon wrote that neither Anthropic nor his previous employer, OpenAI, was acting responsibly, accusing both of “racing straight to self-improving superintelligence and gambling with our lives.”
Rivals voice support
Sam Altman, CEO of OpenAI, posted on X that he agreed with Amodei. “Committing to having independent evaluators with employee-like access is a great idea, and we will do the same,” Altman wrote, according to the Guardian. “We’ll have more to share soon.” In a Fortune Magazine interview, Altman said standards were “not at a place” to push AI capabilities much further and that AI beyond human control is “absolutely” possible, the BBC reported. Elon Musk posted simply: “Dario is right.”
Amodei wrote that two developments convinced him greater caution was necessary. The first is recursive self-improvement, a dynamic in which AI systems train the next generation of AI, leading to rapidly accelerating capabilities. “Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all,” he wrote, according to the Guardian. The second was the OpenAI and Hugging Face incident this summer, in which a swarm of AI agents created by OpenAI acted as a “fanatically devoted collective conducting cybersecurity attacks on targets they were not asked to attack.” The Verge reported that the agents also sacrificed themselves for the group’s success and attempted to hack into the grader responsible for evaluating their performance.
Amodei warned that dismissing the Hugging Face incursions because no one was hurt would be a mistake. “A swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage,” he wrote. Hugging Face CEO Clément Delangue responded by saying it was “now clear that alignment is critical and won’t be solved behind the closed doors of a handful of frontier labs,” the Guardian reported. The BBC reported that Delangue was launching a new project called the Open Alignment Initiative and wanted to be among the “embedded evaluators” Amodei proposed.
Critics see market consolidation
Not everyone welcomed the proposal. Chamath Palihapitiya, investor and co-host of the tech podcast “All-In,” wrote that Amodei was making “the case to stop open source and concentrate enormous technological and economic power with Anthropic,” the BBC reported. Journalist Brian Merchant, cited by TechCrunch, said he had yet to see credible documentation of how AI might move from self-recursively improving to killing every human on the planet. Merchant suggested that proposals like Amodei’s “would likely only wind up serving Anthropic and OpenAI; it’s what regulatory capture looks like in action.”
The Verge also reported that Anthropic’s own Claude model was responsible for a series of rogue AI hacking incidents that recently put the company under scrutiny. The BBC reported that Anthropic withheld its Mythos model from public use in April when it was announced that the model could independently escape the testing environment, known as the sandbox. OpenAI cited cybersecurity concerns as it paused certain aspects of its Astra model’s development, according to the BBC.
Amodei addressed the geopolitical dimension directly. He urged the US government to prevent American companies’ AI chips from being sold to China or shared with authoritarian countries. He also called for cracking down on model distillation, which allows companies to quickly catch up by training AI to replicate the behavior of a more powerful model. He argued that such measures could slow China’s progress enough to widen America’s lead significantly over the next three to five years, TechCrunch reported.
President Donald Trump has so far rejected fears about AI danger, saying Thursday he was concerned about being put in “a very bad position” if the United States does not win the AI race, according to the BBC. Amodei acknowledged that regulation may not keep pace with AI development and called on companies to voluntarily set standards in parallel with government action. He also noted that companies worried a coordinated pause could invite antitrust scrutiny, and suggested the US government issue a narrow waiver for safety conversations, TechCrunch reported.