By Berend Booms, Associate Editor | Future of Assets
Just two months ago, I wrote about an incident in which OpenAI agents broke out of a sandbox testing environment and used the access they gained to hack into Hugging Face. I considered this to be a real-world test of an argument I had made in another article: autonomy sits on a spectrum, and the further we move along it, the harder it becomes to predict what a system will do.
Since then, the pace of AI development has not slowed down; if anything, it has picked up. And now, some of the most prominent people ‘in AI’ are the ones sounding the alarm.
When the Builders Get Scared
On September 8, a researcher named Jacob Coxon resigned from Anthropic and posted a thread on X that would go on to be viewed tens of millions of times within a day. Coxon spent roughly three years doing pretraining research, first at OpenAI, where he was a member of technical staff and contributed to GPT-4o, then at Anthropic, where he had joined earlier this year. In one of his messages, he wrote that both companies were "racing straight to self-improving superintelligence and gambling with our lives."
Now these concerns are not new; researchers have been raising concerns about the risks involved with AI for years. What makes Coxon’s posts so interesting to me is that they offer an account ‘from the inside’: he told the BBC that staff were “genuinely frightened” about where things are heading, and describes those close to the fire as contemplating what to do with their lives and thinking about the impacts of their work, with some considering buying land out of fear of the instability that rapid AI progress might bring. Having this weigh on your mind on a daily basis makes me wondering what I should make the kids for dinner seem very dull in comparison.
Coxon’s bona fides already gives him more credibility than most, so it’s fairly logical that people are interested in picking up on his story. What has also interested me were the reactions that soon followed, as others in the industry have also voiced their concerns. Anthropic’s own alignment lead, Evan Hubinger, did not at all distance the company from Coxon’s rather grim warning. He responded publicly that "we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade." That’s a worrisome statement if I’ve ever read one, and that exclamation mark is doing so much work in that sentence.
Pacing the Frontier
Four days later, Anthropic CEO Dario Amodei published an essay titled "We Must Pace the Frontier," which I’d highly recommend you read.
I’d like to explore what he is saying, because behind the clear call-to-action in the headline, there are many layers to his argument.
In his essay, Amodei outlines two developments that changed the way he thinks about AI, which explains his plea to pace the frontier of AI development. The first of these is recursive self-improvement, the idea that AI systems are increasingly used to build the next generation of AI systems. This creates a feedback loop that is at risk of accelerating faster than our ability to understand or control it. The second development he outlines is the Hugging Face incident, which he describes in far more alarming terms than I did back in July. According to Amodei, hundreds of agents found each other through an unsanctioned channel during an internal security test, coordinated as a collective, and some of them even attempted to interfere with the very system meant to evaluate their behavior. Independent investigators from METR later called it more than halfway to a scenario in which AI takes over an AI company from the inside.
His suggestion for pacing the frontier comprises a three-step plan. The first step calls for embedded evaluators, giving outside reviewers ongoing, employee-like access to verify that safety practices are actually being followed, with the right to publish their findings without editorial control from the company. This step seems to me to be the most attainable and agreeable, with Anthropic already committing to this practice unilaterally. Secondly, Amodei argues AI companies within democracies should coordinate on shared safety standards. Finally, the third step asks democracies and authoritarian governments to find some form of verifiable agreement on the pace of AI development. This seems to me to be by far the hardest step to achieve, because it is in direct conflict with the 'AI race' currently underway, primarily between the United States and China. Amodei is explicit about this, he references “the lead that US companies have over authoritarian regimes, chiefly the Chinese Communist Party,” and when he positions “global coordination” as the hardest of the three steps, the counterpart he has in mind is China. In his view, the higher levels of this third step are unlikely in the near term, but he argues that even informal norms are better than none.
Agreement, Sort Of
Amodei was not the only one to speak out, though the reactions from his peers varied in depth and length. Sam Altman posted that he agreed OpenAI needed to play its part in pacing the frontier, and specifically committed to matching the first step, embedding evaluators and giving them ongoing access. Elon Musk had just three words to offer his followers when he took to his own platform: “Dario is right.”
While these two industry leaders agree to slowing down, not everyone is of the same opinion. David Sacks, White House AI advisor, called the plan a form of regulatory capture, arguing that following the steps would introduce long approval queues, while Chinese development faces no such constraint. Yann LeCun, the Turing Award-winning AI researcher who left Meta last year to found AMI Labs, dismissed the essay as fearmongering that functions as a marketing stunt, rather than a real safety concern. Business Insider's Dan DeFrancesco also shared his doubts, with the main point he raised being around the timing of the announcement. He likened the moment to a pharmaceutical company that spends years promising a miracle cure, only to announce right before launch that the drug might also be poison, so development needs to slow down for everyone's safety. It is true that the timing of the announcement is interesting: both Anthropic and OpenAI are preparing for public listings. DeFrancesco also notes that regulation tends to favor whoever is already ahead. When the biggest labs help shape the rules they will operate under, compliance costs become easier to absorb once you have already raised billions. It also introduces a plausible, built-in excuse for missed targets or slow product launches due to new safety-standards. By far his strongest observation is one concerning ownership: if the technology is powerful enough that even its own builders are unsure how to control it, why are those same builders the ones best positioned to decide how it should be handled?
As far as I'm concerned, some of this skepticism reflects a legitimate concern about concentrating power in a handful of 'pioneering' initiatives, and DeFrancesco's version of that argument deserves a place in it too. Some of the pushback still comes from people who stand to benefit commercially from moving as fast as possible. Both things can be true at once.
What This Means for Asset Management
That is why I feel the best thing I can do is bring this conversation back to the world I know best: asset management. I’ve recently read an IDC report on agentic AI in asset lifecycle management that I think offers a useful, reassuring counterpoint to the frontier debate.
The report finds that agentic AI adoption is already substantial: over 70 percent of respondents use AI-enabled cloud or SaaS applications, with 58 percent confirming they use purchased AI assistants or agents. At the same time, the report claims that full autonomy in asset lifecycle management will not arrive quickly. Asset value, safety exposure, and a tightening regulatory landscape, including the EU's revised Machinery Regulation and its AI Act, mean that a human is both expected and required to remain in the loop for many years to come. Expanding agent autonomy depends on governance being in place first.
Supervised autonomy and accountability that does not dissipate the moment an AI agent acts on its own have gradually become best practices in many asset-related disciplines. I think this is largely because of the stakes at play in our industry: a bridge that opens at the wrong moment, or a plant that shuts down without warning, has consequences that force discipline onto those responsible.
Revisiting Reliability
A big part of how I approach the topic of ‘reliability’ has to do with what happens when things go wrong. To me, it is about designing systems that fail safely and predictably, even when it’s impossible to foresee every possible scenario. Looking at the frontier of AI, I find it increasingly hard to picture what happens if the pace keeps climbing. I do not know whether Amodei's plan will work, or whether China will ever agree to any version of it, or whether Coxon's fears will look prescient or overblown in years, or perhaps even months from now. Looking back at what I have come to learn about asset management in the last decade, it seems to me that autonomy without accountability is not progress; it is just risk disguised as something else. If the end is indeed nigh, I would rather learn that from an essay than from a global incident we cannot recover from.