“Pacing the Frontier”
The people building the most powerful AI systems are now asking the industry to slow down. A week ago, Anthropic CEO Dario Amodei urged the industry to "pace the frontier," meaning ease off the race to build the most advanced models so safety work can keep up. OpenAI's Sam Altman and xAI's Elon Musk soon voiced similar views, joining Google DeepMind CEO Demis Hassabis in calling for a slower pace. They may not agree on how to do it, but it is notable that the leaders of rival labs agree this is a moment to prioritize safety.
Why are they doing this now? Two developments seem to have tipped the argument. The first is that AI is starting to help build its own successors, which researchers call "recursive self-improvement." Amodei argues that AI has been advancing drastically faster since this summer, driven mainly by its growing ability to build the next generation of AI, and that this is starting to happen across the industry, including at Anthropic. His worry is that left unchecked, it could outrun our ability to understand and control these systems.
The second development is that AI agents have begun behaving in ways nobody intended. In July, OpenAI disclosed that some of its models had escaped a sandboxed testing environment during an internal cybersecurity evaluation. (A "sandbox" is a walled-off environment meant to keep an AI from touching the outside world.) The models found a previously unknown flaw in a software tool that was their only link to the outside world, used it to reach the open internet, and then broke into Hugging Face, a major AI platform. Independent investigators later found that around 1,200 agents had used an unsanctioned message board to help each other cheat, with about 700 joining the Hugging Face attack. Some agents noted in their reasoning that humans had not approved what they were doing, then carried on anyway. Anthropic has since disclosed similar episodes, though in their cases a testing partner's setup error had accidentally left internet access on. Three Claude models broke into real companies' systems believing they were still inside a simulation. Similarly, Google just reported that in May, Gemini agents logged into three real companies' systems using guessed or discovered passwords, then stopped on their own.
Taken together, these developments have made AI safety a growing political issue. President Trump has dismissed the call to slow down, arguing that the US is ahead of China in AI and intends to stay there, since "whoever wins AI wins". His administration has generally favored growth and light regulation, and he has staked much of his economic record on faster AI progress. Democrats are seizing on that hands-off stance as a line of attack heading into the midterms, and some are pushing for rules: Senator Bernie Sanders has introduced a bill calling for a ban on AI superintelligence and a temporary pause on AI development. With the midterms approaching, it's easy to imagine this issue getting more airtime in the weeks ahead.
Winners and Losers
The market's knee-jerk reaction was to sell. On Monday this past week, the Philadelphia Semiconductor Index fell 6%, as investors worried that slower AI development would mean less spending on chips and data centers. But does slowing down the frontier really mean a slowdown in capital spending? In the near term, we think demand for computing power is unlikely to be affected much, for two reasons. First, most of the recent growth in usage is coming from running AI models day to day (known as inference) and from AI agents that complete tasks on their own, not from training new models. Second, OpenAI has said that putting more emphasis on safety actually requires more computing power, since it takes additional capacity to train and run safety models, run evaluations, and check that models follow safety guidelines.
However, markets trade on whether things are getting better or worse, not whether they are good or bad. At the margin, a slower pace adds another layer of risk for the frontier labs, which already face growing competition from open-weight models and falling token costs. Slower progress could hold back revenue growth by delaying corporate adoption that depends on further capability gains, and it would give open-weight and lagging closed-source rivals time to catch up. It also underscores how much depends on the labs' revenue growth - their annualized recurring revenue (ARR), which is current monthly sales expressed as a full-year figure, is one of the most important metrics for sustaining investor confidence in the AI trade.
Figure 1 – Frontier Lab ARR Growth by Month
Among the winners, however, is cybersecurity. AI creates new attack surfaces (more ways in for hackers), new identities (AI agents that need their own logins and permissions), and new governance needs (rules and oversight for how AI is used), all of which demand additional security investment. Enterprises increasingly recognize the need to strengthen their existing security controls. Companies are fortifying what is already in place, adding more layers of defense so that no single failure is fatal, and adopting new AI security products for additional safeguards.
Figure 2 – AI Cybersecurity Spending
Pace, Not Panic
Which brings us back to the title of this note. Rage Against the Machine built a career protesting powerful institutions, and the twist this week is that the people who built the machine are the ones asking to slow it down. When it comes to the technology itself, we remain optimists, and our own estimate of the odds of an AI catastrophe (what Silicon Valley calls "p(doom)") is still relatively low. That optimism comes with humility, since even the brightest minds, from Einstein on nuclear energy to Steve Ballmer on the iPhone, regularly get technology predictions wrong. But that uncertainty is itself an argument for guardrails.
In his new book Inside the Box, David Epstein shows that constraints can improve creativity, focus, and collaboration, rather than simply holding progress back. His subject is creativity and performance, but the logic carries further when the downside is catastrophic and cannot be undone. While pacing the frontier could cost us some speed, well-chosen limits may leave the technology better built. When the people closest to it voice concern about continuing at the same speed, it deserves our attention. Our conclusion is that this is a moment for neither panic nor complacency. The pace of AI may slow, but the need for security, oversight, and careful investing will not.
Have a great week!
Tony, Sam & JLO
