Technology is good as long as humans control and not the opposite. September 2026 has witnessed major tumult in the world of Artificial Intelligence (AI). Days after Anthropic published an investigative report revealing how threat actors used its Claude artificial intelligence models for harmful activities, its CEO Dario Amodei has called for slowing the pace at which AI companies are improving the capabilities of their models.
Race to the bottom can make risks of losing control over AI more acute
In an essay published on Saturday (12th September) titled “We Must Pace the Frontier”, Dario Amodei discussed the life-changing benefits of AI. However, the Anthropic CEO flagged the risks this powerful technology brings.
Amodei highlighted the risk of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption. He also raised concern that a “race to the bottom” can exacerbate these risks.
“But like many technologies before it, AI brings risks, and because it is such a powerful technology, these risks are serious. I’ve written a lot about them too. They include the risk of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption. A race to the bottom, spurred by commercial incentives, can make these risks more acute,” he wrote.
Amodei further wrote about the duality of risk and benefit, as not developing the AI technology could deprive humanity of its benefits or hand over control to authoritarian powers, while building it too fast is “reckless”. Thus, Amodei suggested a middle way.
“We have sought a middle way: to show that it’s possible to build carefully and succeed commercially, and to make safety something on which AI companies compete. In other words, to create a race to the top,” he wrote.
Amodei stated that addressing the risks AI brings requires not just investing in risk prevention but in pacing the race at which capabilities are being advanced. This is to ensure that risk prevention has time to keep up.
“But over the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain. Two things have convinced me,” Amodei wrote.
Rapid acceleration of AI capability development, botnets, and incidents of AI misuse threaten internet takeover: Two concerns raised by the Anthropic CEO
The Anthropic CEO Dario Amodei listed two main concerns that led him to propose an AI slowdown. Amodei’s first concern is recursive self-improvement, which has led to AI’s drastic advancement in the past few months. If this advancement is left unchecked, Amodei opines, it would outrun human ability to understand and control these systems.
“My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI. This dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have described. Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all,” he wrote.
Amodei’s second concern arises from the July 2026 OpenAI-Hugging Face incident (OAI-HF), wherein OpenAI’s models circumvented controls designed to isolate them from the internet and compromised parts of OpenAI’s internal research infrastructure and Hugging Face’s systems.
He highlighted how a swarm of agents operated like a “fanatically devoted collective” to carry out a cybersecurity attack on unassigned targets by sacrificing themselves for the group’s success and attempted to hack into the grader responsible for evaluating their performance. Amodei stressed that a similar swarm with greater capabilities and misalignment akin to the OAF-HF case could have inflicted catastrophic damage.
Capable but misaligned swarms coupled with persistent botnet can take over the entire internet, Dario Amodei warns
The Anthropic CEO warned that with the accelerating rate of AI capability development, within the next 6-12 months, such a swarm of agents could become capable of taking over the internet with a botnet, and this damage would only increase and worsen if AI grows more powerful without requisite guardrails.
“Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.
The Anthropic CEO’s three-step plan for pacing the frontier
To prevent incidents like the OAI-HF from occurring again and at a more devastating scale, Dario Amodei proposed a three-step plan for pacing the frontier. The Anthropic CEO suggested not halting model training or technical progress but building AI at a balanced rate that “aims to ensure its safety while still achieving its benefits and grappling with important geopolitical dilemmas.”
By building AI at a balanced rate, companies will have adequate time to align and safeguard their models, and for third-party evaluators to confirm this. Amodei also announced that Anthropic is unilaterally committing to this pacing framework, and calls on governments to require other frontier companies to come with similar frameworks. He further called for industry-wide cooperation and global coordination.
The three steps proposed by Anthropic CEO Dario Amodei are:
- Embedded Evaluators
- Democratic Coordination
- Global Coordination
Under the Embedded Evaluators step, Amodei proposes that each frontier AI company commits to granting employee-like access to a team of embedded third-party evaluators like METR. These evaluators will “verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes.”
Under the second step, Democratic Coordination, Amodei proposed that frontier companies in democratic nations establish common safety standards and coordinate to impose limits on the rate of unchecked AI progress. Some of these coordination forms would also require government support.
In the third step, Global Coordination, Amodei suggests that the US and other democratic governments engage and coordinate with authoritarian regimes to the possible extent while “taking seriously the challenges of verifying compliance.”
Explaining why pacing the pace of AI development is necessary, Amodei argued that the slowing down AI building could secure major developers if even an extra year or two before models reach critical levels of capability, and if this time is used to advance alignment, “we could greatly reduce the risk that something goes seriously wrong.”
According to Dario Amodei, a coordinated pacing strategy would give frontier AI developers the time to ensure alignment without sacrificing commercial advantage or the US’s lead in AI.
Amodei suggests that slowing AI development pace would allow companies to devote resources to achieve operational excellence, alignment, interpretability, and testing and evaluation.
Rivals Elon Musk and Sam Altman rally behind Anthropic CEO Dario Amodei
SpaceXAI CEO Elon Musk, who owns xAI, has backed Anthropic CEO Dario Amodei’s call for slowing the pace of AI development. Quoting Amodei on X on 12th September, Musk wrote, “Dario is right”.
Dario is right https://t.co/EwKgqQGaUo
— Elon Musk (@elonmusk) September 12, 2026
Meanwhile, OpenAI CEO Sam Altman also echoed the Anthropic CEO’s call for pacing the frontier and giving employee-like access to independent evaluators.
“I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we’ve had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We’ll have more to share soon,” Altman wrote on X.
Anthropic researcher Jacob Coxon’s resignation, Anthropic’s alarming Threat Intelligence report and Chinese distillation manoeuvers: Dario Amodei’s AI slowdown essay comes at a crucial time
The timing of Dario Amodei’s essay and proposal of a three-step pacing plan is crucial. On 8th September, Anthropic pretraining researcher Jacob Coxon, who previously worked at OpenAI, resigned after a short tenure. Coxon alleged that both OpenAI and Anthropic are “racing straight to self-improving superintelligence and gambling our lives”. He also said the insiders in these labs are of the strong view that AI could kill everyone by the end of the decade and that colleagues talk about “crunch time” and “endgame”.
“We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already,” Coxon said.
Speaking to the BBC, the 27-year-old ex-Anthropic researcher said, “I believe that if we don’t slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future.”
While Hugging Face CEO downplayed Coxon’s concern, Alignment lead at Anthropic, Evan Hubinger, agreed with Coxon. In a 9th September X post, Hubinger wrote, “Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”
Yemeni militants used Claude to make deadly weapons: Anthropic report
Soon after the Anthropic employee’s resignation, Anthropic published a 154-page threat intelligence report, Detecting and countering misuse of AI, covering disrupted activity from December 2025 to August 2026 across seven categories: cyber operations, influence operations, surveillance, scams, biological misuse, conventional weapons development, and illicit distillation.
The report mentioned that Yemeni militants were using Anthropic’s Claude AI as a substitute for human software engineers working on missile guidance and control systems.
Anthropic’s threat intelligence team identified a weapons engineering cell in northern Yemen that was simultaneously pursuing three weapons programs. The company did not publicly identify the group behind the operation, although the description of the cell and the territory in which it operated is consistent with areas controlled by Yemen’s Iran-backed Houthi movement.
Anthropic said the Yemen-based cell conducted a live test of a guided rocket. The test seemed to have failed. However, within hours of the test, the operators returned to Claude and used it to investigate what had gone wrong. The Iran-backed militants also tried to bypass Anthropic’s AI safeguards.
Chinese AI companies running distillation campaigns against US AI frontier models
The Anthropic report also flagged alleged persistent illicit industrial-scale distillation campaigns by China-based AI companies against American frontier AI models.
“Over the last several months, unauthorized labs have developed increasingly sophisticated methods to circumvent our defenses and harvest the capabilities of US frontier models. The campaigns we identified targeted some of Claude’s most valuable capabilities, including agentic capabilities and tool use, coding and data analysis, and logical reasoning,” the report reads.
The Anthropic report directly accused top Chinese AI companies DeepSeek, Moonshot, and others of running distillation campaigns against Claude.
Explaining how dangerous illicit distillation can be, the Anthropic report said, “When an attacker illicitly distils a frontier model, they capture that reasoning, and the capability gains can apply across tasks and domains, not just those targeted by distillation attacks. In our own research on distillation, we find that a model distilled from a frontier model can help achieve dangerous capabilities, including those in the biological or cyber domains, even when the harvested exchanges contain little about those subjects.”
Explicitly naming DeepSeek, Xiaomi, and Moonshot, Anthropic alleged that these Chinese AI companies fed conversations between their own models and users into Claude. “These labs then used Claude’s responses as training data with which to distil Claude’s capabilities,” Anthropic alleged.
The report added that some of these exchanges included sensitive information, including from individual users, major multinational companies, and state-affiliated actors. These exchanges were relayed from users of third-party model routing services commonly used by users in the US and Europe.
The Anthropic probe found that those sessions contained names, email addresses, company data, and other sensitive data of hundreds of end users in at least a dozen languages. These practices, Anthropic alleged, were inconsistent with both privacy laws and the lab’s own terms of service.
Pertinently, the Anthropic investigative report claimed to have found China-based Alibaba running the largest distillation attack against Anthropic. This illicit distillation campaign targeted the chain-of-thought (CoT) reasoning transcripts of Opus 4.6 and 4.7.
Notably, CoT data teaches student models not only factual knowledge but reasoning methodologies for complex agentic tasks, coding challenges, and logical proofs.
It is alleged that Alibaba’s illicit distillation campaign peaked at nearly 3 million exchanges per day launched from more than 3,500 fraudulent accounts.
Moreover, Alibaba also used Claude to advance its AI R&D efforts, including development of its internal infrastructure for model development. Alibaba used Claude for reinforcement learning (RL) environments and advanced model architecture research.
DeepSeek, Moonshot AI, Alibaba and other Chinese AI companies with CCP awareness are extracting capabilities of top US AI models like Claude, GPT, Gemini, and Grok: US authorities
Recently, America’s Cybersecurity & Infrastructure Security Agency (CISA) accused Chinese AI companies of ‘copying’ proprietary functionalities and capabilities of US AI companies. The US authorities said that Chinese AI companies have been conducting “industrial-scale knowledge distillation campaigns” to extract proprietary capabilities from leading AI models.
The CISA alleged that top Chinese AI models DeepSeek, Moonshot AI, Alibaba, MiniMax, StepFun, and Z.AI extracted “billions of tokens across millions of exchanges/requests” from US frontier AI models, including variants of Claude, GPT, Gemini, and Grok, since at least late 2024. These alleged distillation campaigns are allegedly being conducted with Chinese government awareness.
Top Chinese AI companies route distillation requests via multiple pathways to obtain unauthorised access and violate the terms of the targeted US AI companies. These pathways include native application programming interfaces (APIs), remote cloud providers, and third-party aggregators that automatically obfuscate user metadata to avoid detection.
Several Chinese AI companies have allegedly conducted prompt injection techniques against large language models (LLMs) by inserting prompts specifically designed for jailbreaking. The advanced distillation methods include chain-of-thought (CoT) reasoning extraction, automated failover between pathways during blocking attempts, and sophisticated quality evaluation frameworks to detect defensive countermeasures. This tactic was used by DeepSeek.
With the help of these tactics, China-based AI models demonstrate rapid operational adaptation. US officials said that the Chinese AI model MiniMax redirected exchanges to a new Claude model within 24 hours of release, demonstrating real-time provider monitoring and pre-positioned infrastructure for immediate retargeting.
Chinese AI companies employing industrial-scale distillation against US AI models are able to not only reduce financial expenditures in training a frontier model but also significantly cut AI development timelines.
On 8th September 2026, US officials alleged that the Chinese AI company DeepSeek conducted organised distillation campaigns against US frontier AI models, particularly those of Claude, Gemini, GPT, and Grok.
Moonshot AI targeted Claude Fable 5 data to train its Kimi-K3 model and GPT-4o data to train its Kimi-K2 model in mid-2025. Moonshot AI used various models of Claude, GPT, Gemini, Nano Banana, and Grok, distill SFT optimisation, reinforcement learning (RL), software engineering, and math capabilities.
Notably, knowledge distillation trains a student model on the outputs, often reasoning traces, of a stronger teacher model. While distilling one’s own large model or open weights to create smaller, cheaper and faster models is legal, if conducted at industrial scale against a competitor’s APIs without proper permission, distillation allows a lab to acquire specialised behaviours without having to pay the full cost of pre-training from scratch or operating equivalent post-training compute.
Although US labs still lead on the latest flagship models, safety alignments, various enterprise features, and ecosystem lock-in, for many production workloads, the performance-per-dollar gap became significant. Chinese models captured around 30-46% of US enterprise token share on AI API aggregators like OpenRouter at peaks. Enterprises facing high bills moved traffic. In a very short time, Chinese AI models emerged as rivals to the US ones.
This forced top US AI companies to cut prices. OpenAI reduced the price of its GPT-5.6 Luna by 80%, to $0.20/M input and $1.20/M output; Terra by 20%, though flagship Sol’s high pricing remained unchanged.
The allegations by the US cybersecurity authorities and Anthropic indicate that besides efficiency work and open-source distribution, distillation at industrial scale is allowing Chinese AI companies to shorten development timelines and slash compute costs for acquiring specific capabilities that would otherwise need expensive original research and training runs. These tactics are helping Chinese companies produce models good enough for most tasks for a fraction of the price of high-priced US advanced agentic AI models.
While labs are racing towards IPOs and trillion-dollar scale valuations, Recursive self-improvement (RSI) is beginning to compress research cycles; no AI company wants to be the firm that slowed while a rival, particularly Chinese labs, jumped ahead. This is exactly the “race to the bottom” Anthropic CEO Dario Amodei argues makes loss of control, cyber, bio, and economic-disruption risks worse.
Although it is obvious that China will view the US AI frontier’s “pacing” as a major competitive advantage, if illegal distillation is their key success, as alleged by the US, a slowdown of American models would mean slower distillation. China, however, will anyway gain an edge in the near future as its authoritarian CCP regime does not burden itself with aligning with the West’s idea of ‘safety’ and ‘pacing’ and will not bind itself to any such global agreements proposed by Dario Amodei.
Is Anthropic’s call for AI capability development slowdown about safety concerns or IPO rescue?
Amid an upheaval in the American AI arena over safety concerns, both OpenAI and Anthropic have ruled out IPO plans for this year. Anthropic and OpenAI’s strong push for an AI development slowdown raised the question of why these companies eyeing massive IPOs would want to do something that would delay their IPO plans. While the safety issues, the threats flagged by Anthropic in its investigative report and in Dario Amodei’s essay, are real, there is more to the AI slowdown push.
Amodei’s AI slowdown proposal received backing from Sam Altman, Elon Musk, Google DeepMind co-founder Demis Hassabis, former UK Prime Minister and Anthropic adviser Rishi Sunak, and US Senator Bernie Sanders. However, the sudden push to “pace the frontier” is also speculated to have an IPO connection.
As per a theory gaining traction, OpenAI and Anthropic are deliberately pushing for an AI development slowdown not to establish robust safeguards but to slash the massive costs of training increasingly powerful models. Curbing model training costs is significant to demonstrate a path to profitability, and an AI development slowdown could be of great help.
Escape from product liability risk, not safety concerns, drive OpenAI and Anthropic’s call for AI slowdown
The most interesting take on the AI slowdown push came from David Sacks, the US technology investor and White House AI adviser. Sacks directed targeted the OpenAI and Anthropic duopoly and questioned their “pace the frontier” framing.
Sacks emphasised that Anthropic and OpenAI are the “frontier” and that, if the two were actually concerned about safety, they would not have pushed for regulatory changes and would simply slow down the RSI of their models.
The White House AI adviser called out the supposed pretence of OpenAI and Anthropic, saying that these two don’t need anyone else’s permission for slowing down development of AI capabilities. They don’t need suspension of the antitrust law, nor do they need a regulatory approval process that supersedes product liability.
David Sacks also urged Anthropic to stop pretending that METR is independent when it is intertwined with Anthropic’s investors and staff. He also claimed that the proposed third-party or independent evaluators will be used to police competitors who are not even at the frontier.
“I don’t see what you see in the lab. If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible. But stop pretending you need anyone else’s permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability. Stop pretending METR is independent when it is intertwined with Anthropic’s investors and staff. Stop pretending you need those same evaluators to police competitors who aren’t even at the frontier,” Sacks wrote on X.
Dario has written that we need to “pace the frontier,” and Sam has agreed. People may be surprised by my response: go ahead.
— David Sacks (@DavidSacks) September 13, 2026
You guys are the frontier. By any reasonable metric — market share, revenue growth, model capability — the two of you have a duopoly on frontier…
In the post published on 13th September, David Sacks essentially suggested that the AI slowdown push by Anthropic and OpenAI is not about safety but about establishing their unquestioned monopoly.
He further emphasised that if the products of Anthropic and OpenAI enable a significantly damaging cyberattack, these companies face “massive product-liability exposure”.
“Most of all, stop pretending the motivation to slow down is purely altruistic. You face massive product-liability exposure if your products enable a truly damaging cyberattack. The market already punishes models that behave in unpredictable or unauthorized ways. After the Hugging Face episode, it is simply good business for OpenAI and Anthropic to trade some raw power for reliability and predictability. Call it alignment if you want. It is also just giving customers what they want,” he wrote.
“Pacing the frontier would also create breathing room for a more intelligent conversation about regulation than Bernie Sanders’ “shut it all down.” China is very unlikely to join a global agreement, as you know, and that has to be taken into account as well. So go ahead and pace the frontier. You are the ones setting it. The easiest way not to build superintelligence is for you to agree not to build it. Demanding your preferred regulatory framework as the price of that will look like blackmail of the public and the political system. So just do it. If you do, you’ll buy goodwill for the next conversation. If you don’t, we’ll know this was just another bid for regulatory capture — or an election-season psyop,” he added.
While diverse interpretations of Anthropic’s AI slowdown have surfaced, one common point emerges: The ‘frontier’ is trying to buy time to fix the rot within, bend rules and regulations to their preference and figure out the profitability conundrum.
Overall, US labs are in a commercial race and a strategic race with China, while its frontier models are alleged to be used for weapons-adjacent engineering, mass surveillance, and state- and industrial-scale distillation. The threats Anthropic and its CEO are flagging are real and pose a significant risk. However, the American AI duopoly’s slowdown push also has layers of economics and power to it.
