1. When everyone does the right thing
Suppose every AI system involved in a decision is safe. Every company follows the applicable rules. Every engineer does the job properly. Every organization completes its risk assessment.
Is the result safe? Most of us answer yes without thinking about it. If each part works as intended, the whole should work as intended too. That intuition is becoming harder to defend.
Consider urban mobility. A navigation platform optimizes routes for its users. A logistics operator optimizes deliveries. A city uses AI to improve traffic flows. Public transport adjusts service according to predicted demand. Autonomous vehicles select efficient routes while remaining within their safety requirements. Nothing here is wrong. Each system performs the task it was built to perform. Each organization acts within its own responsibilities. Each decision is locally reasonable. Yet the combined result may be one that nobody wanted.
Traffic shifts into residential areas. Competition for road and curb space intensifies. Public transport demand changes in ways that affect service availability. Emergency access becomes harder, energy demand shifts, and eventually so do local businesses and where people choose to live. No AI system has failed. No organization has broken a rule. The state of the whole emerges from the way individually reasonable decisions interact.
Put this way, the problem is clearly not new. Systems engineering has long known that individually sound components can produce unsound systems. Economists call a version of it externalities. Financial regulators learned it in 2008, when every institution could satisfy its own prudential requirements while the system itself became unstable — and the response was not only stricter rules for each bank, but a new layer of oversight concerned with the connections between them.
Business has been meeting the same problem for years. Value is increasingly created across organizational boundaries — through platforms, ecosystems, and services produced in continuous interaction with users and partners — while responsibility stays divided along the older lines. Anyone who has tried to build a platform business, coordinate a supply chain, or run a large industrial ecosystem has met the gap, long before AI was involved.
AI did not create this structure. It is entering it. And as it does, an apparently simple question becomes hard to answer. When something goes wrong, or simply goes badly, who is responsible? Not for one model, and not inside one company, but for what the whole arrangement produced together. The difficulty is rarely that no one is responsible. It is that everyone is, partly, and no one is, entirely.
2. What happens when the actors are AI
Recent research on populations of AI agents offers an unusually clear window into this problem. In experiments published in Science Advances in 2025, Ariel Flint Ashery, Luca Maria Aiello and Andrea Baronchelli studied decentralized populations of large language model agents and found that social conventions emerged spontaneously through repeated local interactions. More strikingly, strong collective biases emerged during that process even when the individual agents showed no bias at all.
Other experiments have placed AI agents in competitive markets. In work by Sara Fish, Yannai Gonczarowski and Ran Shorrer, pricing agents built on language models reached supracompetitive prices and profits in oligopoly settings quickly and on their own, with no instruction to coordinate — and small, apparently harmless differences in how they were instructed changed how far this went. Related work on multi-commodity competition found agents settling into market division.
The underlying intuition is familiar enough. A project with one person, three people, and ten people is not the same project scaled up; coordination, deference, and misunderstanding appear somewhere along the way and change what the group is capable of. Or take a street with a pedestrian, two drivers, and a cat: each may take the safest action available to it, and the combination can still produce a collision. One is not simply a smaller version of many.
Research on multi-agent AI is now examining this difference systematically. A 2025 technical report from the Cooperative AI Foundation, assembled by Lewis Hammond and a large group of collaborators, organizes the terrain into three failure modes — miscoordination, conflict, and collusion — together with seven underlying risk factors, among them information asymmetries, network effects, selection pressures, and destabilizing dynamics.
Within that terrain, several patterns recur. Information is filtered. When content passes along a chain of language models, it is not carried faithfully. Alberto Acerbi and Joseph Stubbersfield found in PNAS that a language model reproducing text through transmission chains favored the same kinds of content humans favor — stereotype-consistent, social, negative, threat-related — over other material. What survives the journey is not what mattered most. It is what travels best.
Small patterns are amplified. As signals move through a population, minor tendencies can grow until a bias exists at the collective level that no individual model exhibits — which is precisely what the Science Advances experiments observed.
Diversity collapses. Ilia Shumailov and colleagues showed in Nature that models trained on recursively generated data degrade in a particular way: the tails of the distribution disappear first, and the rare cases go quiet before the common ones do. It is a slow failure, and a quiet one. Every individual output can remain acceptable while the range of the whole narrows.
Actors become locked in. Agents can settle into coordination patterns that none of them would have chosen alone, and that none of them can leave unilaterally — a failure of coordination rather than of any participant.
These four are not an exhaustive list, and I do not expect them to remain the full picture. Scale itself appears to matter: in the same Science Advances study, a small committed minority of adversarial agents, once past a critical size, could impose an alternative convention on the entire population. Larger systems may not simply show more of the same behavior. They may show different behavior — and as these populations grow in number, in diversity, and in the density of their connections, I would expect more complex collective dynamics than the ones we have so far been able to observe.
Why should any of this happen at all? Because an AI system does not respond according to a fixed rule. Its response varies with the context, with the user, and with whatever it is responding to. Put several of them in contact and each becomes part of the environment that shapes the others’ behavior. A small difference does not stay where it started. It travels, and it is reshaped by every actor it passes through. That is propagation, and it has a security dimension as well. Donghyun Lee and Mo Tiwari have shown that a malicious prompt can self-replicate across interconnected agents much like a computer virus, spreading even when agents do not share all of their communications with one another. Others have demonstrated attacks that travel through the communication channels themselves.
None of this means that AI agents will inevitably collude, become biased, or behave dangerously when they interact. It demonstrates something more fundamental: knowing the properties of each component does not tell us the properties of the system they create together. I have argued elsewhere that AI governance still assumes localized causality in environments now defined by propagation. These experiments show, in miniature, what that shift looks like.
The safety of each individual AI system remains essential. It is simply not sufficient. The question is no longer only whether each AI is safe. It is also what happens when they interact.
3. But the real world is much larger than a multi-agent system
The real world is fundamentally different from a controlled experiment. In an experiment, interactions can be isolated: a fixed number of agents, a defined task, a bounded environment. Outside it, nothing is isolated, and the things an AI system responds to are not only other AI systems. An autonomous vehicle does not interact only with other autonomous vehicles. It interacts with human drivers, pedestrians, cyclists, traffic infrastructure, navigation platforms, insurance systems, manufacturers, regulators, weather conditions, and the physical environment. An AI system used in agriculture may interact indirectly with farmers, machinery, seed providers, chemical suppliers, financial institutions, weather systems, soil conditions, logistics providers, retailers, and consumers. A decision made by one actor alters the environment in which another actor operates. That actor responds, changing conditions for someone else. Effects travel across organizational, technological, geographic, and even temporal boundaries.
Increasingly, the things we care about most, whether a system is safe, whether it is fair, whether it can be relied on, do not belong to any one identifiable “it.” They emerge through interactions among many.
This makes the problem broader than multi-agent safety. The relevant actors may be AI systems, but they may also be people, companies, robots, vehicles, infrastructure, markets, institutions, shareholders, communities, or elements of the natural environment. And they do not all share the same objectives. A technically safe AI can optimize successfully for its user while shifting costs elsewhere. A company can comply fully with its obligations while creating dependencies for another organization. Several locally sensible decisions can reinforce one another until they settle the whole into a state that nobody intended. The more deeply AI becomes embedded in existing systems, the more these interactions matter.
4. So how do we keep such an environment safe, ethical, and fair?
That is the practical question. Many AI systems, including physical ones, interacting with infrastructure, organizations, markets, and people — how do we keep the state that all of this produces within something we would call safe, ethical, and fair?
A great deal of the answer already exists. AI governance has made substantial progress in recent years. Frameworks and regulation increasingly address questions that were once poorly defined: who develops an AI system, who deploys it, what risks must be assessed, what documentation is required, when human oversight is necessary, how high-risk uses should be managed, and how responsibilities should be distributed among different actors.
The EU AI Act is an important example of this evolution. It establishes obligations across different actors and categories of AI, including providers and deployers, high-risk AI systems, general-purpose AI models, and models associated with systemic risks. Other frameworks around the world similarly offer increasingly sophisticated approaches to risk management, accountability, testing, and oversight. These mechanisms remain essential. Nothing that follows replaces them.
But they were built around a particular unit of analysis: a system, a model, an organization, a defined responsible actor. And they rely on instruments — testing, audit trails, incident analysis, the reproduction of failures — that quietly assume repeatability and traceability. Systems whose behavior shifts with context, embedded in arrangements where effects propagate across organizations, weaken both assumptions at once.
Meanwhile, many of the effects that matter most do not belong neatly to one model, one system, or one organization. They arise between them. An individual actor may see its own risks but not the cumulative effects created across a larger network. An organization may have authority to intervene in its own system but not in the dependencies around it. Responsibility may be clearly assigned within each organization while becoming fragmented across organizational boundaries. The challenge, therefore, is not that existing governance has failed. It is that the object we need to govern is becoming larger.
5. So what is it that we would actually govern?
One answer is to place controls at the points where systems meet. Interfaces, contracts, data-sharing agreements, third-party audits. These are useful, and much of it is already done. But if effects propagate, checkpoints at the boundaries will not be enough. By the time something arrives at an interface, it has already been shaped by everything upstream, compressed, amplified, reinforced. The problem is not located at the seams.
What would have to become visible instead is the structure of relationships itself, and the conditions that structure keeps producing. That means asking a different set of questions. Who depends on whom? What information moves between actors, and how is it changed on the way? How does one decision alter the choices available to another? Where do incentives reinforce each other? Who can see the state of the whole? And who has the standing to act on it?
Several existing practices would need to stretch to accommodate this.
Risk assessment would need to consider not only direct harms from an AI system, but how decisions and effects travel through connected actors — including actors the assessing organization has no relationship with.
Monitoring would need to look not only at model performance, but at the condition of the whole over time. Some of the most consequential changes are slow and undramatic: a gradual narrowing of diversity, a hardening of dependencies, a concentration of influence. None of these would ever appear as a failure in any individual system.
Accountability would need mechanisms for situations in which no single actor caused a result, yet several actors collectively produced it. This raises a harder question than it first appears. If the state of a system emerges through interaction, can responsibility still be assigned in advance — or does it also, in part, emerge?
And governance itself would need to be treated as something that continues, rather than something completed at the point of approval. Relationships, dependencies, incentives, and power structures change as actors respond to one another. An arrangement that looks balanced when introduced may become something quite different a few years later, without any single decision having caused the change.
None of this implies governing every interaction, or steering the whole from a central point. In practice it may be closer to the opposite: intervening as little as possible, while deliberately maintaining the conditions under which a healthy state can sustain itself. Those conditions are themselves a design question, whether diversity is preserved rather than gradually optimized away, whether decisions remain reversible, whether actors can still enter and leave, whether anyone can see enough to notice when the state is drifting. That is a different design problem from making each component trustworthy. Both are necessary.
6. From healthy AI to healthy systems
This distinction will become more important as AI moves deeper into healthcare, mobility, manufacturing, finance, energy, cities, and other domains where value is created collectively rather than by one technology acting alone. It may also help explain why some forms of AI adoption are easier than others.
When AI produces a relatively self-contained output, organizations can usually define the product, the user, the risk, and the responsible actor with reasonable clarity. When AI becomes embedded in complex operational systems, value and risk increasingly depend on interactions that cross those boundaries. The challenge ahead is therefore not simply to build more capable AI, or even to make every individual AI system responsible.
We need ways to see the structure they form together — how decisions propagate, how incentives interact, how dependencies form, how influence accumulates, and how responsibility shifts as that structure evolves — and ways to keep that structure in a condition we can live with. This is what I refer to as ecosystem governance.
The difference is not one of scope but of object. Existing governance asks whether each participant is behaving properly. This asks whether the whole they form is in a condition worth sustaining, even though no one owns that whole, and no one was ever asked to answer for it. It is not a replacement for AI governance, regulation, safety engineering, or organizational accountability. It is a way of connecting them when the thing we care about can no longer be understood through any one of them alone. Because every component can be working as designed. Every actor can be following the rules. And the whole can still move in the wrong direction.
Healthy AI doesn’t guarantee a healthy ecosystem.
2026/07/27 Mari Sekino
References
Acerbi, A. & Stubbersfield, J. M. (2023). Large language models show human-like content biases in transmission chain experiments. PNAS, 120(44). https://doi.org/10.1073/pnas.2313790120
Flint Ashery, A., Aiello, L. M. & Baronchelli, A. (2025). Emergent social conventions and collective bias in LLM populations. Science Advances, 11(20), eadu9368. https://doi.org/10.1126/sciadv.adu9368
Fish, S., Gonczarowski, Y. A. & Shorrer, R. I. (2024). Algorithmic Collusion by Large Language Models. arXiv:2404.00806.
Hammond, L. et al. (2025). Multi-Agent Risks from Advanced AI. Cooperative AI Foundation, Technical Report #1. arXiv:2502.14143.
He, P., Lin, Y., Dong, S., Xu, H., Xing, Y. & Liu, H. (2025). Red-Teaming LLM Multi-Agent Systems via Communication Attacks. Findings of ACL 2025, 6726–6747.
Lee, D. & Tiwari, M. (2024). Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems. arXiv:2410.07283.
Lin, R., Ojha, S., Cai, K. & Chen, M. F. (2024). Strategic Collusion of LLM Agents: Market Division in Multi-Commodity Competitions. arXiv:2410.00031.
Sekino, M. (2026). From Localized Causality to Propagation Dynamics: Rethinking AI Governance Assumptions. SSRN Working Paper.
http://dx.doi.org/10.2139/ssrn.6776720
Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R. & Gal, Y. (2024). AI models collapse when trained on recursively generated data. Nature, 631, 755–759. https://doi.org/10.1038/s41586-024-07566-y
Regulation (EU) 2024/1689 of the European Parliament and of the Council (Artificial Intelligence Act).
This essay outlines a problem rather than solving it. The hardest part of that problem is one I have deliberately left open here: an ecosystem has no natural edge. Deciding where one ends — which actors are inside it, whose effects count, what state we are trying to keep healthy — is already a governance act, and at present no one holds the authority to perform it. That question, and what follows from it for responsibility, observation, and design, is the subject of a paper in progress.
This line of thinking has developed through conversations with Prof. Ibo van de Poel (TU Delft) since June 2025, on responsibility and ethics in platform ecosystems — a question I brought to him from business practice, and one that begins before, and extends beyond, AI. The arguments here, and their shortcomings, are my own.


