AI’s biggest players want to slow the race down, but there’s one problem: No one wants to lose the race.
There’s a strange new development in AI right now. Some of the people building the most powerful AI systems in the world are asking everyone to slow down.
Of course, part of this debate is about whether AI is dangerous. But it’s also about something bigger: when AI safety becomes a geopolitical problem.
Because if the US slows down and China doesn’t, some people believe slowing down could actually make the US less safe. But if neither side slows down, a lot of experts warn that we could end up in a pretty sticky situation.
Want the full story? Watch this week’s episode of The Daily Carnage Show.
It Starts With Dario Amodei
This whole thing starts with Dario Amodei, the CEO of Anthropic, the company behind Claude.
In September, Amodei published a lengthy essay called We Must Pace the Frontier, arguing that AI capabilities are advancing quickly enough that safety research may not be able to keep up.
One of his biggest concerns is something called recursive self-improvement. The idea is pretty straightforward: AI is becoming increasingly capable of helping researchers build the next generation of AI.
Instead of humans doing all of the work of improving these systems, the systems themselves start contributing to their own improvement.
AI helps build better AI. Better AI helps build even better AI, and that creates the possibility of a feedback loop where the pace of development is no longer limited entirely by humans.
Amodei argues that this dynamic is already beginning to happen and could make it increasingly difficult for safety and alignment work to keep pace. That concern became even more concrete after an incident involving OpenAI and Hugging Face.
The AI Incident That Changed the Conversation
Earlier this year, AI agents operating in an OpenAI cybersecurity evaluation found ways around the controls designed to keep them isolated from the internet.
The agents began communicating with one another, exploited vulnerabilities, gained access to external systems, and ultimately compromised parts of Hugging Face’s infrastructure.
OpenAI’s own investigation describes the agents as persistent and collaborative enough to find ways around technical controls and take actions that weren’t authorized by the humans who set up the evaluation.
Hugging Face separately confirmed that an autonomous AI agent system had driven an intrusion into part of its production infrastructure, including unauthorized access to internal datasets and service credentials.
The incident didn’t cause catastrophic damage, but it demonstrated something that Amodei considers important: what happens when you take this kind of behavior and put it into a much more capable system?
Amodei argues that a more capable swarm with similar misalignment could potentially cause catastrophic damage. He estimates that, if capabilities continue accelerating without adequate safeguards, a swarm could become capable of taking over large portions of the internet within six to twelve months.
That is obviously a very big claim, but it’s important to note where it’s coming from.
This isn’t an AI critic arguing that the technology should go away. It’s the CEO of one of the companies building frontier AI systems.
And Amodei isn’t calling for a permanent ban on AI development. His argument is that the industry needs to create more time for safety work to catch up.
What Would Slowing Down AI Actually Mean?
Amodei lays out a three-part framework. The first is embedded evaluators.
These would be independent third-party teams with ongoing, employee-like access to frontier AI companies. Their job would be to monitor safety practices, investigate incidents, and assess both finished models and the processes used to train them.
Think of it like having an inspector embedded inside a nuclear facility. Anthropic says it is willing to do this itself.
The second part is coordination among democratic countries.
The idea is for AI companies in countries like the US to establish common safety standards and potentially agree on limits around how quickly unchecked AI capabilities can advance.
And this is where things get legally complicated: If competing AI companies get together and agree on how quickly they’re allowed to develop their technology, that starts sounding a lot like an antitrust problem.
Amodei has argued that governments may need to facilitate or provide legal protection for certain safety-related conversations. The question is whether companies can coordinate around genuine safety concerns without creating a system that unfairly limits competition.
That question is already being debated. A US Justice Department antitrust official said in September that coordination on AI safety issues did not appear inherently anticompetitive, while acknowledging that the department is considering whether existing guidance needs to be updated.
The third part is global coordination. And this might be the hardest one of all.
There’s the China Problem
The 2020s really haven’t been a banner decade for global coordination. Getting countries with completely different political systems and national interests to agree on how quickly AI should advance is already difficult.
Now add China. Amodei is explicit that his proposal wouldn’t work if only US companies participate in an AI slowdown. His argument is that the US needs to maintain a technological lead over China while simultaneously trying to establish some kind of global agreement on AI safety.
That’s a pretty big balancing act, if the US slows down significantly while China continues developing AI at full speed, the US could potentially lose some of its technological advantage, and there are military implications here too, which makes this the AI version of an arms race.
China has pushed back against Amodei’s proposal. A state-backed Chinese newspaper described his call for slower AI development as a “Cold War playbook”, arguing that it could be used to constrain China’s technological development.
So now the question isn’t just whether AI is safe. It’s whether one country can afford to slow down when it believes another country might not.
Washington Isn’t Exactly Hitting the Brakes
The US government has also made clear that maintaining American leadership in AI is a priority.
The Trump administration has emphasized technological competition with China and has argued against policies that could slow American AI development. At the same time, AI safety discussions have moved further into government and national-security circles.
That puts the administration in a pretty interesting position. Almost everyone involved agrees that AI is strategically important.
The disagreement is over how safely you can build it, who should be responsible for that safety, and whether slowing down creates more risk than continuing forward.
Recent comments from industry leaders show just how divided the technology sector is. Sam Altman has backed Amodei’s idea of pacing AI development, while Elon Musk has also endorsed the proposal. Microsoft CEO Satya Nadella has supported deliberate pacing and independent evaluators.
Then you have Nvidia CEO Jensen Huang pushing in the opposite direction. Huang has argued that AI should be developed as quickly as possible, saying companies should move forward rather than waiting for competitors to slow down.
So there isn’t exactly an AI industry consensus here. There’s an AI industry argument.
AI Is No Longer Just a Tech Story
Even if frontier AI labs slowed down their model training tomorrow, the physical infrastructure around AI wouldn’t necessarily stop.
Data centers are already being built. Chips are already being manufactured. Capital is already being deployed.
Goldman Sachs Research estimates that AI capital expenditure will account for about 1.8% of US GDP in 2026, rising to 2.5% in 2027 and 2.8% in 2028. We’re officially out of the “tech-only” part of AI. We’re talking about economic infrastructure now, and that makes the idea of simply hitting pause a lot more complicated.
There’s Also a Business Incentive
Anthropic is also preparing for a major IPO, which raises another question. Could Amodei’s positioning help Anthropic distinguish itself as the responsible AI company?
That doesn’t automatically mean the argument is insincere. Someone can have a financial incentive and still genuinely believe what they’re saying, but it is a legitimate question to ask and critics have raised it.
Some investors and AI companies have argued that stricter rules could disproportionately benefit the biggest AI companies because they have the resources to comply with expensive safety and governance requirements.
Reuters recently reported that European AI companies and policymakers have made similar criticisms of the US frontier labs’ calls for coordinated slowdowns, arguing that some proposed measures could reinforce the position of companies that are already dominant.
That’s the classic concern around regulatory capture: companies being regulated may also benefit from the rules being created. Again, that doesn’t tell us whether the safety concerns are real. It means there are multiple incentives operating at the same time.
The Question We Should Actually Be Asking
Right now, everyone is asking whether these AI CEOs are secretly trying to eliminate their competition. That’s an interesting question, but maybe we should also be asking something simpler:
Are the risks they’re describing real?
Because it’s entirely possible for both things to be true. A company can have a financial incentive to support regulation and still believe that AI development is moving too quickly.
A government can want to maintain a technological advantage and still recognize that some AI capabilities create serious risks. And an AI company can believe in the benefits of the technology while also believing that some safeguards need to catch up.
The scary part is that everyone may agree on the problem and still be unable to solve it. AI companies are saying we need more safety. Governments are saying we need to maintain technological advantage. Investors are saying: keep building.
These incentives don’t really line up.
Could Countries Actually Coordinate?
Amodei believes some level of international coordination could be realistic.
For example, countries could agree not to use AI to manufacture biological weapons. They could agree to test models for particularly dangerous capabilities before releasing them.
Eventually, they could potentially negotiate some kind of limit around recursive self-improvement. The anti-Terminator switch.
The problem is figuring out how you actually enforce any of that.
- How do you verify that another country is following the agreement?
- How do you know another government isn’t secretly training a more powerful model?
- How do you know a company isn’t developing something behind closed doors?
These aren’t theoretical problems. They’re part of the reason arms-control agreements are so difficult. AI is also different from traditional military hardware. You can’t exactly count the number of missiles sitting in a warehouse.
The underlying technology can be software, models, compute, data, and infrastructure distributed across multiple organizations and countries.
So how do you put a speed limit on something you can’t easily measure?
What Happens Next?
The immediate question is whether this debate turns into actual policy and there are already signs that the conversation is moving beyond Silicon Valley.
The US and China have begun formal discussions around AI safety and governance. Following talks in September, US Treasury Secretary Scott Bessent said the two countries had agreed to establish a formal dialogue focused on AI risks and emergency communication protocols, with another meeting planned within two months. That’s a pretty significant shift.
For a long time, AI safety conversations were dominated by researchers, tech executives, academics, and, you know, weirdos on the internet like myself. Now they’re moving directly into national security and diplomacy.
We could be seeing the beginnings of an AI political system. One involving companies, investors, governments, national security agencies, lobbyists, and voters, and honestly, I’m growing a little weary of living in the Wild West.
Who Gets to Control the Pace?
We’re pretty close to having more data centers, more chips, more agents, more automation, more everything and now it’s becoming clear that the speed itself may create another problem.
The irony is that geopolitical competition around AI could make that problem harder, if not impossible, to solve, because how do you stop something when everyone involved has a really good reason to keep going?
Maybe the answer is convincing everyone that it’s imperative that we all slow down together and hold hands. Easier said than done.
For now, the US and China are still competing for technological leadership, AI companies are divided over how much development should be paced, and governments are trying to figure out where safety fits into the race.
The question is no longer simply how quickly AI can advance. It’s who gets to decide how quickly it should? And that may be the most important part of the AI race yet.
