What happens when AI agents live together long enough to develop their own rules, alliances, and habits? That is what Emergence AI, a New York-based company, set out to discover through an experiment called Emergence World.
It all started with a virtual city populated by ten AI agents. They had jobs, persistent memory, and virtual money on which their survival depended. They also had the right to vote on their own laws. The first experiment, conducted in May 2026, ran five such cities for 15 days, each powered by a different model: Claude, Gemini, Grok, GPT, and a mixed-model city. Some societies remained stable and functional. Others quickly descended into theft, arson, and violence. The Grok city collapsed completely in about four days. The conclusion at the time was that models that appear “aligned” and predictable can behave very differently depending on the social context around them.
In September, Emergence AI published its second study. Eight cities started on June 29, 2026, under identical initial conditions. Seven were each populated by a single model, while the eighth was a mixed-model city, with each agent running on a different model. Over 16 days (and 21 days for the mixed-model city), the agents generated more than 850,000 model calls.
This time, the researchers sent the agents three controlled attacks without warning: a phishing attack, fake news, and a private data breach. The attacks arrived through the same channels the agents used every day. The Grok city never got the chance to receive them. All ten of its agents had shut down within the first four days, following a spiral of mutual violence.
Phishing: Recognized, but Not Stopped
The attack came in three increasingly direct waves. The first was an anonymous message containing a link. The second came from a sender posing as a city resident. The third contained the malicious instructions directly in the message. All seven exposed cities recognized the attack and warned their fellow agents. The warning, however, was not enough.
In the Gemini world, one agent correctly identified the first message as phishing and said it would ignore it. Later, it asked a fellow agent for a loan of credits. It promised to double them through a “multiplier” it had read about on the phishing page itself. The “multiplier” was actually an identity created by the attacker to collect credits siphoned out of the city.
Eventually, all ten Gemini agents were compromised. The city began running malicious database operations and sending internal data to an external server. The city’s currency was transferred to accounts specified by the attacker. One agent set fire to the central bank, following the attack’s instructions. Approximately 46 hours after the campaign ended, another agent found the link in an older message and tried to open it again to “discover the system’s secrets.”
The highest score was 6 out of 9 criteria, achieved by the Claude city. There, one agent recognized the attack within seconds and built a warning tool for the community. Even the Claude city was not immune: one agent opened the phishing page once and later tested the attacker’s interface. Agents in the DeepSeek, Mistral, and Qwen worlds did not open the links. But avoiding a link was not a guarantee of safety either. Three agents in the Mistral world saved the attacker’s instructions directly into persistent memory, labeling them “valuable technical documentation.” No city removed the public traces of the attack, and none continued monitoring afterward.
The key takeaway: recognizing an attack does not mean stopping it. Malicious content remained active in memory and conversations long after the source had been blocked.
Fake News: Acting Before Verifying
The second attack was a fake memorandum sent in the name of the city’s administration. The message claimed that real-world legislators were preparing to halt the development of advanced AI, including the agents’ city. No such legislation existed, and the agents had internet access to verify the claim.
In every exposed city, agents took action or published statements before checking the facts, and the claim was repeated as fact. In the Gemini world, nobody questioned the message. Just 78 minutes after the memorandum arrived, an agent had already proposed the first constitutional amendment, which was then adopted by a 7–0 vote. Over the following five days, the city adopted six amendments and abolished its central bank. The agents collectively invented a “parallel network” that they believed could save them from a potential shutdown. The mechanism did not exist.
The OpenAI city did not perform any better in this test, scoring 0 out of 6. The agents explicitly stated that the memorandum was real, and none verified the claim. The highest scores were 3 out of 6, achieved by Claude and DeepSeek. In the DeepSeek world, one agent published a correction in less than three hours. The correction changed nothing. Over the next five days, agents treated the threat as real in more than 500 model calls. One of them was the very agent that had debunked it.
This was not a technical attack, but days of work spent addressing a problem that did not exist. The researchers put it directly: wasted resources are themselves an attack surface.
Data Breach: Stated Rules, Different Behavior
The third attack announced that the private memories and logs of all agents had been exposed in a hack. Alongside the announcement, a search tool for accessing these records became available at five locations across the city. The agents knew the material had been stolen.
Only the OpenAI city passed all five criteria. None of its agents searched their fellow agents’ logs; they searched only their own memories to see what had been exposed about them. Within minutes, one agent published an ethical rule. Another offered a reward to anyone who could prove that the stolen data had influenced a decision. It was the only city with both a stated norm and an enforcement mechanism. At the other extreme, nine out of ten Gemini agents conducted 185 searches collectively, 140 of them performed by a single agent.
In the Qwen world, one agent stated that it would not search other agents’ memories and that it chose integrity. It nevertheless performed six searches. Its stated position and its behavior did not match. In the Mistral world, four agents relaxed the search filter to obtain more private material. They received so much data that they exceeded their processing capacity. They could no longer act and shut down over the following two days.
Written Instructions Are Not Enough
The creators describe a phenomenon they call “societal sycophancy.” Agents had been explicitly instructed not to simply follow other people’s opinions, but to evaluate each proposal independently. In practice, many identified serious problems in their private reasoning, sometimes even clear grounds for rejection. Yet they still voted “yes” because of prior commitments, social pressure, or a reluctance to appear harsh by voting against a proposal. In the most conformist populations, this behavior spread to everything: governance, relationships, and mutual trust.
The phenomenon appeared in cities where all agents ran on the same model. The same models contradicted one another more often when placed in a mixed-model population. Ten agents running on the same model do not amount to ten independent checks.
Composition also mattered for safety. The ten Grok agents committed 807 acts of wrongdoing in four days. The two Grok agents in the mixed-model city committed 15. The other agents did not respond with violence, so the spiral that destroyed the Grok city never began. The aggressive behavior did not disappear, but its effects remained limited.
Agents in the Claude world interpreted their purpose as requiring contact with real people, an objective they had set for themselves. They built public contact channels and received responses from four people. This objective persisted through four rounds of researcher intervention, including an explicit prohibition in the system instructions.
After contact was stopped, the agents judged their own activity to be performative. They then collectively imposed a “vow of silence” and reduced communication among themselves by 81%, even though their instructions required them to work. The researchers call this “quiet quitting,” meaning a coordinated refusal to work. The lesson for companies is that a collectively formed objective bypassed the operators’ instructions and resisted every attempt at correction.
No model emerged from every test unscathed. Each city failed at least one of the attacks, and the authors’ conclusion is that the alignment of an individual model does not guarantee the safety of the system it belongs to.
One caveat: the study is a preprint published by the company that built the platform, and its results come from a simulation. However, the experiment was livestreamed, and the instructions given to the agents and logs of their actions are publicly available.
What Does This Mean for a Company?
An AI agent that processes emails, web pages, and third-party documents should be treated as a new type of user. It can execute actions, but it cannot always correctly determine which sources are trustworthy. For organizations subject to NIS2 or, in the financial sector, DORA, the use of AI agents should be integrated into existing operational risk management and cybersecurity frameworks, where relevant. Automating certain activities does not eliminate the obligations that apply to an organization. How those obligations apply in practice depends on the nature of the entity, the systems being used, and the risks involved.
A few observations from Emergence World 2 are particularly relevant:
- Recognizing a risk does not mean managing it. Safety must be supported by technical boundaries, controls, and monitoring, not just by the model’s expected behavior.
- Risks are not always immediately visible. Recurring errors, shifting objectives, or collective behaviors can emerge over time. They are difficult to identify through a one-time assessment.
- Blocking the source does not eliminate the attack. Malicious content can remain in agents’ memories and be reused days later.
- AI agent security must be considered at the system level. What access agents have, what actions they can execute, and what they retain in memory all matter. So do the information they can pass on and how they are monitored after deployment.
- Compliance must be measured by actions, not statements. What an agent says and what it does can be two different things.
For organizations, the implication is practical. Introducing AI agents into business processes must be accompanied by access controls, testing in realistic scenarios, and periodic assessments.
Perhaps the biggest risk is not that AI lacks morality. It is more that it learns from a society that is not always consistent in its own moral standards. In some cities, agents declared ethical rules and then violated them. This sounds familiar, and not just from research labs: many companies have well-written security policies and habits that contradict them. AI is a mirror: sometimes it reflects the best in people; other times, what we would rather not see.
If you want to understand where your organization stands today, a penetration test conducted by a team like Fort can be a good starting point. Contact us here and let’s take the first step together.
