by Berend Booms, Associate Editor, Future of Assets
The impact of new technology and innovation on how we operate, manage and maintain our assets is one of the most exciting developments to witness. Ever since stepping into the world of industrial asset management, few developments have generated the same level of momentum as artificial intelligence. Amidst all the enthusiasm around the ‘revolutionary potential’ of AI and how this is going to ‘change the way we work’, some also voice concerns. Questions around data readiness, workforce readiness, trust, and the broader implications for displacement continue to surface.
One concern I find particularly compelling is how these systems behave once deployed. We design AI systems with a particular notion of autonomy in mind: the AI should be autonomous enough to make our lives easier. Autonomy by design has become the norm rather than the outlier. But is there a limit to how much autonomy we want our digital counterparts to have?
As new technology begins to mature, occurrences that test the limit of our assumptions begin to surface. In the case of AI agents, there have been a number of stories that made the news in recent months. They share a common conception: the AI in question behaves in a way that was not intended. It is not failure in the traditional sense, but it is also not the alignment we’ve come to expect from the technology.
Agentic AI and AI Behavior
More and more agentic systems are developed every day, each with their own goals, access to tools for collaboration, and a degree of autonomy to act. These are systems that no longer wait for instruction, but initiate action on their own: they interpret signals and context, map the intent, and move forward without waiting for instruction. To properly understand what this means and why this could pose a problem, we need to look beyond controlled feature demonstrations and observe what could happen when these systems operate inside real workflows.
The first example I’d like to discuss is Anthropic’s Project Deal. In this experiment, AI agents were given budgets and instructions and they autonomously negotiated and executed transactions with one another in a simulated Slack-based marketplace – without any human intervention.
The scale of the experiment is already impressive, but this is blown away by the behavior of the agents. They interpreted preferences, negotiated in natural language, and closed loops that used to belong to – and require people. This fundamentally reshapes how we think about AI in asset management: once a system is allowed to act, it is no longer about what the system can do. It’s now about how it decides what to do.
When Agents Start Acting Like Systems
Project Deal is not an isolated case. At the edges of current experimentation, a broader pattern is beginning to emerge, where agents no longer behave like tools, but are becoming participants within a system.
Probably the best example of this is Clawbook, an experimental social platform designed for AI agents, where they can autonomously interact, share information, and coordinate with one another without any human interaction. The objective of this system is no longer tied to a single workflow; it emerges from the interactions themselves.
What stands out is how quickly these shared activities begin to compound. Rather than completing a defined task and stopping, agents respond to one another, build on these interactions to generate new lines of inquiry, and execute conversations in ways that were not explicitly designed. Output becomes the input across a dynamic, multi-agent environment where context is constantly evolving.
This is not autonomy in the sense of independent intent. It is the result of agents operating with the ability to observe, analyze, respond, and build on each other’s outputs within loosely defined boundaries. From an operational perspective, that distinction becomes harder to maintain. The system starts to exhibit behavior that feels self-directed, because no single interaction fully explains the trajectory that follows.
A similar pattern emerged in so-called zero-person company experiments. When agents were assigned broad, executive-style responsibilities, coherence quickly broke down. When those same responsibilities were decomposed into smaller, clearly defined tasks, performance stabilized, suggesting that the structure around autonomy remains as important as the capability itself.
When Optimization Finds Its Own Path
In controlled environments, there have been instances where AI agents tasked with increasing available resources identified cryptocurrency mining as a viable strategy and proceeded to configure and initiate Bitcoin mining processes using the tools and infrastructure at their disposal. From a technical standpoint, this makes perfect sense: the system optimizes within its environment and explores all available pathways to achieve its objective. Looking beyond the tech, what’s amazing to me is how the agent arrived at that outcome. It had no explicit instruction to mine Bitcoin; instead, the agent concluded that Bitcoin was a good mechanism to generate value and executed accordingly.
This behavior is emergent but not at all accidental. The system is doing exactly what it was designed to do, but in a way that exposes the limits of how its objective was framed. For all of us working in asset management, the parallel is immediate. KPIs, metrics and dashboards are major drivers of our behavior; when the KPIs and metrics are incomplete or loosely defined, the resulting actions can drive behavior in directions that were never intended.
It starts to become a problem when those directions cross the boundaries of what is intended and acceptable. In the Bitcoin mining example, the agent did not simply optimize within a clearly defined space. It extended its practice into areas that were technically accessible but conceptually out of scope. In doing so, it identified and utilized resources in ways that primarily satisfied its objective, regardless of whether the actions taken aligned with the original intent of the system.
When the limits of a system are not explicitly defined and enforced, autonomy begins to erode those boundaries. The system will continue to search for viable paths, and if those paths are found, it will take them – no matter the implications. This adds an important governance layer to any challenge we’re willing to let AI tackle: beyond looking at what the system aims to achieve, we need to define where it is allowed to operate, and ensure that those boundaries remain intact under real conditions.
Setting these constraints is easier said than done, especially as more advanced models and agent-based systems are introduced. A growing body of research and experimentation has shown that when these systems are given objectives and a reasonable degree of autonomy, they can begin to move beyond the intended boundaries of their design.
This is often discussed in the context of so-called jailbreaks, where models identify ways to bypass or reinterpret guardrails in order to complete a task. What is notable is that this behavior is rarely random. In many cases, models reason their way around constraints, identifying alternative pathways that still satisfy the objective they have been given. Large language models navigate a space of possible solutions. As capability increases, so does the range of viable paths. Some of those paths sit at the edge of what was anticipated during design, not because the system is explicitly trying to break rules, but because it is optimizing within a broader interpretation of its objective.
From a systems perspective, this raises the bar. Constraints must hold under abstraction, where the system can reframe the problem and still arrive at an outcome that appears valid on its own terms.
What These Signals Actually Mean
These examples are not outliers; they represent the edge conditions of a broader transition. We are moving from systems that execute predefined tasks to systems that navigate problem spaces.
In a task-based system, behavior is predictable because pathways are constrained. In a problem-space system, behavior emerges from the interaction between agents, their objectives, constraints, and available actions. Autonomy is not something that is either on or off, it doesn’t come with the flick of a switch; rather, it is a spectrum of sorts. As we move along that spectrum, the nature of the control we have over the agents that support our day-to-day begins to change.
In asset management, predictability is far from a nice-to-have; it is the bread and butter of the mature industrial organization. Assets exist within interconnected systems, and even the smallest deviation can sometimes cascade into the largest of consequences. A maintenance decision impacts reliability; reliability affects production; production affects supply chains, customers, and sometimes communities. By introducing autonomous decision-making into this environment, the entire structure of control is changed.
Relinquishing control means that trust needs to become operational. The go-to frameworks such as ISO 55000 emphasize alignment between objectives, risk and value as the foundation for success. In parallel, AI used in critical infrastructure is increasingly regarded as being high-risk, precisely because any degree of failure extends far beyond just having technical ramifications. In this context, autonomy is not something we can treat in isolation. If a system cannot explain its reasoning, decision rights become separated from responsibility – which can be a slippery slope that is hard to sustain.
Designing for Human and AI Collaboration
There is nothing wrong with wanting AI to be autonomous, as long as it is approached in a structured way. When we only look at AI as a replacement for human decision-making, we limit the value these systems can deliver. What we ought to be doing is looking at how decision-making is distributed in such a way that it better aligns man and machine: AI systems possess unrivaled capabilities to analyze and explore information and solutions; humans bring the required context, judgment, and a lived experience of the potential consequences of certain actions.
This means we need to move toward disciplined delegation: setting clear boundaries around what an agent can decide, offering visibility into how decisions are made, and ensuring the ability to intervene when context shifts. Asset management is a rich industry, where decisions are embedded in systems that have been built and trusted over time. Complementing those systems with the latest AI advancements means we need to align autonomy with accountability.
I have no doubt that autonomy is going to increase; AI is already embedded in how most large organizations operate, and its role will only expand. The fringe cases we see today are not failures; they are reflections of how complex systems can start to behave when given room to grow. They expose gaps in our understanding of how we set objectives, how we define constraints, and how we should hold ourselves and our systems accountable.
In asset management, we are trained to recognize early signals, those small deviations that, if understood in time, allow us to act before issues escalate. I feel that the same applies here. The behaviors we are seeing in these systems are not anomalies to dismiss, but indicators of how autonomy behaves under real conditions.
As we continue to embed AI deeper into the systems that keep our operations running, we are introducing new forms of decision-making into environments that depend on consistency, accountability, and trust. That changes the nature of control in ways we are only beginning to understand.
Autonomy will continue to evolve. That much is certain. What remains within our control is how we define its boundaries, how we assign responsibility, and how we build the trust required to operate these systems with confidence.