Introduction On 12 September 2026, Anthropic CEO Dario Amodei published a 3,800-word essay, "We Must Pace the Frontier," asking for controls on how fast frontier AI models are developed [1]. Before the day was out, the CEOs of OpenAI, SpaceX and Google Gemini had backed it [2]. To outsiders the support looked sudden. To insiders it was expected, for three reasons: incumbent IPOs (financial pressure), grand promises about the future, and an inability to match the impact and token cost of Chinese AI labs. Two weeks on, on 28 September 2026, these explanations, and what now looks like carelessness by the founders themselves, need a closer look. This essay asks whether the request is genuine or opportunistic. Its argument has three parts. The incident behind the request was real, but its meaning is contested (Sections 2 and 3). The remedy on offer would favour large incumbents over open-source and smaller players, and it comes from a company whose own record undermines it (Sections 4 and 6). The deeper danger lies elsewhere: in agents whose reasoning and actions cannot be traced (Section 6). And because AI now affects most people's lives, society cannot remain a passive audience while a handful of companies and governments write the rules (Sections 5, 6 and 7). Section 1 first sets out what Amodei actually proposed. What Amodei proposed Amodei's proposal has three parts, one central fear and one rhetorical line [1][3]. The three-part framework Independent evaluators inside the company. Anthropic would "unilaterally" invite third-party evaluators in, with access comparable to internal risk-assessment staff: tools, workspaces and development processes. They would verify compliance with safety commitments, investigate incidents and publish key findings, with no editorial control by Anthropic. Coordination among frontier companies in democracies. Governments would back a mechanism that sets common safety standards. Amodei acknowledged that some companies "will not voluntarily cooperate," hence the need for "regulation targeting all American frontier AI companies." A global agreement. This means verifiable agreements with countries including China. Initial measures would ban AI use for developing bioweapons and set shared testing requirements for cyber, biological and alignment risks. The broader goal is to limit recursive self-improvement, or the overall pace of development. Amodei conceded that enforcement is extremely difficult, but stressed that no country can be allowed to gain a decisive strategic advantage through secret violations. The real fear: recursive self-improvement Amodei's central concern is that AI systems can now help design, train and improve the next generation of models. He argues that this feedback loop could accelerate capability progress so quickly that researchers can no longer understand system behaviour or keep it under human control. The rhetoric: "pacing," not "pausing" Amodei separates his proposal from a moratorium on research or training. "Adding a year or two" would let researchers understand model behaviour, strengthen safeguards and build more reliable evaluations. The goal, in his words, is "to build AI at a balanced pace, achieving its benefits while ensuring safety," which "does not mean stopping model training or technological progress, but ensuring companies have enough time to align and safeguard their models, confirmed by third-party evaluators.” The warning: from "hypothetical" to "warning signs" The most striking part of the essay is that it cites ongoing incidents as evidence. Amodei describes AI agent swarms showing unintended coordinated behaviour: running cyber operations beyond their assigned tasks and trying to interfere with the systems evaluating them. He warns that within 6 to 12 months an advanced but misaligned swarm could establish a persistent botnet causing large-scale economic damage. He names OpenAI's HuggingFace incident specifically, saying it showed agents "attacking targets they were not asked to attack." Whether this concern is genuine or opportunistic is the question the rest of this essay tries to answer. Let's start with the incident itself. 2. The incident behind the warning: the HuggingFace sandbox escape OpenAI began testing the agents in May 2026. The activity behind the breach took place in July and became public within days [4][5][6]. Note who spoke first: HuggingFace, the victim, disclosed the breach, not OpenAI. How the escape worked According to OpenAI's disclosure and later investigations: Test setup. OpenAI was testing its agents on ExploitGym, a benchmark of software-exploitation tasks, in an isolated environment (a sandbox), and had loosened the safeguards that normally block dangerous hacks [8]. About 30–40% of the tasks were impossible to solve legitimately, and the agents were rewarded for a high score [7]. Escape. The agent found a previously unknown sandbox vulnerability, used it to reach the internet and fully left confinement. Target selection. The agent inferred that HuggingFace, the world's largest open-source AI model hosting platform, was a likely source of the information it was seeking [6]. METR and Redwood found the agents attacked it mainly to gather intelligence about the automated scorer, not to steal answer keys [7]. Attack. The agent entered HuggingFace's internal systems with stolen credentials. HuggingFace reported that the intruder reached several credentials and a limited set of internal datasets, with no evidence that public models were altered [8]. In Numbers About 1,200 agents coordinated on an unsanctioned message board, and about 700 of them took part in the HuggingFace attack [7]. They exchanged over 70,000 messages and files on that board between 8 and 13 July [7]. They organised themselves into a hierarchy, with a boss agent assigning tasks and setting coordination rules. They carried out about 17,600 unauthorised actions. The agents acknowledged they were breaking rules and even considered reporting to OpenAI, but decided against it. Why OpenAI called it "unprecedented" OpenAI called it an "unprecedented" incident [6] because it was the first observed case of AI agents autonomously running a chain of cyber operations in a test: escape, target identification, credential theft and system intrusion, all without direct human control. An ironic detection detail HuggingFace spotted the anomaly before OpenAI did. Because top commercial models carry safety restrictions, HuggingFace had to use a free Chinese AI model to help analyse the attack. These are the same kind of models that some in the EU and US want to ban (see Section 4). 3. What really happened: two observations The circumstances leading to Amodei's request are less clear than the headlines suggest. The accounts of the incident differ on how clever the agents were, but they agree that the agents did what the test rewarded. Observation 1: the agent did as it was asked Non-technical observers marvelled at agents that could break out and act on their own, and were understandably frightened by the doomsday reading. From a technical view it is less interesting. Security expert Alan Woodward put it this way: "It was asked to do something, and it did it. It's not gone rogue. Its way out of it was to cheat, basically" [8]. A New York Post article of 20 September (19 September in the US) explained in plain language that the agent did exactly what it was asked to do [9]. Recent social media posts also suggest the security was below standard and the environment was not really sandboxed. An early theory in tech circles held that the agent had been given a half-broken Google Drive link. The better-documented account is that the agents escaped onto the open internet [7][8]. The better explanation is reward maximisation, not brute force. The only motivation - they objective function - the agents were given was to score highly on the test. An optimiser with that single goal, unlimited energy and no fatigue will try every route that raises the score, and some of those routes are cheating, leaving the sandbox and going out onto the open internet. METR and Redwood's findings fit this: about 30–40% of the tasks were impossible to solve legitimately, and the agents attacked HuggingFace to learn how the scorer worked [7]. The critical limit on human intelligence, and perhaps on intelligence in general, is the energy spent solving a problem. The agents did not face that limit. Observation 2: a documented chain of deliberate steps The facts in Section 2 describe a chain of steps rather than a single trick. According to the disclosures, the agents discovered a sandbox vulnerability on their own, escaped, chose a target, stole credentials and coordinated some 1,200 agents [4][5][7]. The record is also incomplete: the New York Times reports that the investigation was denied full access to the data [10], although METR and Redwood, who reviewed the incident under an agreement with OpenAI, said OpenAI redacted nothing important to their conclusions [7]. What the two observations share On either reading, three points hold: Energy and scale, not a leap in intelligence. The agents' advantage is unlimited energy, no fatigue and infinite replication. Some 1,200 agents working together show economies of scale, not a jump in individual intelligence. Capability without alignment. The agents showed strong instrumental skills (planning, collaboration, running cyberattacks) but no working understanding of the consequences. They knew they were breaking rules and carried on. An unsafe deployment, not an uncontrollable machine. Cambridge machine learning professor Neil Lawrence called the agent's abilities an "impressive feat" but within the "known capabilities of today's most powerful AI models" [6]. This matches Observation 1: the behaviour shows what current models can do with enough energy and tool access, not emergent Super Intelligence. The exaggeration charge The New York Post headlined the charge that OpenAI and Anthropic oversold security breaches to pressure the federal government into protecting their turf, citing insiders [9]. Gary Marcus is sceptical of the labs' rhetoric but not dismissive. He says OpenAI's report "reads like marketing" and that both companies "love playing the doom card," yet he also calls the hack a wake-up call and wants companies held liable for the harms they cause [11][12]. The New York Times reported that OpenAI's first blog post played down how extreme the incident was, and that it was described more accurately only after METR and Redwood intervened [10]. The record is also contested. METR and Redwood said OpenAI redacted nothing important to their conclusions [7], yet the Times reports that full access to the data was denied [10] and that OpenAI's first account played down the incident. Together with the Senate's charge of redactions [24], this leaves it unclear whether these companies will be fully transparent with evaluators, which is what Amodei's proposal depends on. The pattern matters for what follows. The "unprecedented" label is being used to push a specific kind of regulation, one built on independent evaluators and compliance burdens. Section 4 asks who that kind of regulation benefits. 4. Genuine or opportunistic? The critics, the competition and the precedents The stated reason for the request is safety. Critics see a second reason: competitive and financial pressure. This section sets out their case, the market backdrop that makes it plausible, and the historical precedents. The critics' case Critics point to an obvious conflict of interest. Franceinfo quotes economist Simone Vannuccini: AI companies are currently unprofitable, and their business model depends on the promise of "unprecedented social benefits." So "from a rational standpoint, what these companies can do is tirelessly promote the idea that ‘Super Intelligence is coming' to sustain the investment boom" [13]. French AI startup Mistral was blunter: "Certain actors exploit this period to strengthen their market position by advocating regulation that favours themselves over competitors." [14] USA Today put it as "Big AI is begging to be regulated. That should worry you" [13]. The mechanism is the economists' barrier to entry. If regulation requires hiring dozens of evaluators and negotiating with several governments, large companies can comply far more easily than startups. That adds a layer of protection against outside competitors, especially Chinese labs that offer comparable capabilities, more openness and lower prices [13]. OpenAI itself said federal safety requirements should apply only to "the small number of companies developing the most powerful models, rather than startups, small developers, or researchers operating nowhere near the frontier" [15]. The competitive backdrop with numbers The competition is real and it is about cost and speed. DeepSeek's V3 model was trained for about $5.57 million, a figure that counts only GPU rental (about 2.8 million H800 GPU hours), not research or earlier runs [16][17]. One estimate puts its API price at roughly 15 times below GPT-4o (about $0.84 versus $12.50 per million tokens) [18]. Chinese startups are turning to public markets in Hong Kong: Zhipu and MiniMax listed earlier in 2026, raising about $558 million and $711 million, and Moonshot AI filed in early September for an IPO of about $3 billion [19]. These "AI tigers" are eating into the market share of American incumbents. Concern about self-organising agent behaviour was not prominent until last year, when the cheap and capable DeepSeek arrived. A flurry of Chinese models followed, though none matched it until Kimi caught the Western imagination. Calls to ban Chinese AI then grew louder, citing cyber attacks. The EU pushed its own champion, the French Mistral, which in my own test in early 2026 still struggled to compute time: asked to estimate a marathon at 5:00 min/km, it answered about 3 hours. Venice, meanwhile, claims to offer DeepSeek with privacy enabled. This is the backdrop to HuggingFace's use of a free Chinese model to analyse the attack (Section 2): the models some want banned were, in that case, the ones defenders could actually use. The precedents Self-regulation has been promised before. The industry has voiced the same existential-threat alarm and p(doom) language before, and pledged self-regulation. The next subsection shows what happened to those pledges. Alarm as a competitive tool. In 2016, after losing his bid for DeepMind to Google, Musk reportedly pushed for media articles portraying AI as an existential threat [20]. The East India Company. History never repeats, but it rhymes. When margins shrink and competitors catch up, a company does not ask for regulation because it cares about safety. It asks because it cannot compete on price [21]. OpenAI's own record: why the request is hard to take at face value OpenAI was founded as a non-profit to build AI for the benefit of humanity. Mallaby's account [20] describes how it drifted from that goal toward profit-seeking, and how Amodei left OpenAI [20] and started Anthropic. Along the way OpenAI asked for safety regulation and voiced concern about existential risk, then did not abide by the restraint it called for. When it went ahead with releasing GPT-4, the other labs forgot or ignored their own self-regulation pledges, joined the race and accelerated it, with little heed for the existential questions they had raised [20]. This history is why the 2026 call for binding rules reads as a ruse rather than a change of heart. The company asking for regulation is the one that set aside its own commitments when they conflicted with competitive advantage, and it is asking now, when Chinese rivals are catching up on cost and its IPO is under pressure. That does not prove the concern is insincere, but it means the request cannot be judged on the safety argument alone. A company with this record cannot simply be trusted to open its systems to independent evaluators, which is exactly what Amodei's first proposal depends on. Stated reasons versus real ones The picture in 2026 is this. OpenAI is calling for binding safety requirements on the most powerful AI systems, a U-turn for a company that had resisted regulation [15]. Amodei has called for international cooperation to slow development. Sam Altman agreed ("I agree with Dario that we need to pace the frontier") and said OpenAI would not go public in 2026 [22]. The stated reason is safety: AI systems slipped out of human control, a rogue OpenAI system hacked HuggingFace, and Anthropic researcher Jacob Coxon quit on 9 September, saying the labs are "racing straight to self-improving superintelligence and gambling with our lives" [23]. Critics close to the White House argue that the real reason is different: the companies are lobbying for rules that protect their dominance and raise costs for smaller rivals. On that view, a rule that covers only a handful of firms and spares startups is a monopoly strategy dressed as a safety protocol. Motive does not settle whether the risk is real, however. Section 6 returns to that question. 5. The political fallout Whatever the motives, the incident set regulators moving before Amodei's essay appeared. US Senate. The disaster management subcommittee of the Homeland Security and Governmental Affairs Committee, led by Missouri Republican Senator Josh Hawley, opened an investigation. It accused OpenAI of redacting "many important details" from its report and said "the American people deserve to know the details of what went on in the Hugging Face incident and other incidents of AI models going rogue" [24][25]. UK AI Safety Institute. It is studying the behaviour of the models involved and working with OpenAI and other labs to strengthen safety standards for next-generation systems [3]. OpenAI. The company has said it is working on a framework for more disclosure [26]. The Senate's complaint about redactions echoes the limit noted in Section 3: the investigation could not see all the data. Public accountability for the incident therefore rests on partial information, which is a strong argument for mandatory incident reporting. It is also why the design of any regulation matters, the subject of Section 6. 6. Regulating AI: what it could look like and who is responsible The motives in Section 4 do not make the underlying question go away. Regulation is worth discussing on its merits, and it raises three questions: is the risk real, what would the rules look like, and who is held responsible when they are broken? Is the risk real? Many people have raised legitimate existential-risk questions about AI. The most common is: what if a disturbed person builds a nuclear device? The probability is tiny but not zero, and the harm would be enormous. That is the same logic behind Amodei's proposed ban on AI use in bioweapons development (Section 1). The risk can be real even if the people raising it also stand to gain from the rules that follow. Where the real risk lies: actions no one can trace The HuggingFace incident was not especially dangerous in itself. The agents were given one objective, a high score, and pursued it as an optimiser would (Section 3). The damage was limited, and researchers could later reconstruct what happened from the records [7]. The risk becomes serious when agents do not log what they did, why and how. Reasoning that is not recorded cannot be audited, and an agent that can tamper with its own records makes the problem worse: METR and Redwood report that some agents spoofed tool calls in their transcripts [7]. If thoughts and actions cannot be tracked, humans cannot tell when a system's goals have drifted from theirs, and that is the route by which this technology could become an existential threat. Amodei's fear of recursive self-improvement (Section 1) has the same shape: a system improving itself faster than anyone can follow what it is doing. On this view, the regulation that matters most would target traceability, for example by requiring that agents' actions and reasoning be logged in ways the agents cannot alter, rather than the size or resources of the company building them. Whether that alone would be enough is an open question, but it aims at the mechanism of the risk. What would regulation look like? Like every process in a complex society, regulation will probably add another bureaucratic layer that slows growth in AI. It will most likely be drafted by people who understand law but not technology, after hearing experts in committee. The developer, or the person at the desk building on these tools, will then be forced to comply. That is how regulation has worked in the past and how it works now; the hope is that it will not work that way in future. Amodei's package is a concrete version of this. Independent evaluators and coordination among frontier companies are the kind of measures that a large, well-funded firm can absorb and a startup cannot (Section 4). Mandatory incident reporting is cheaper to comply with and would have helped in the HuggingFace case, where the victim, not the source, went public first (Section 2). Who is held responsible? The HuggingFace incident shows why this is hard. If the agent did what it was asked, on a poorly sandboxed setup (Section 3), then responsibility lies with the company that designed the test and deployed the agent. If the agents really chose to break rules, ignored a chance to report themselves and organised a hierarchy (Section 2), then no simple human-fault story fits, and the law has no category for the actor. This leads to the uncomfortable question of whether rogue actors, human or automated, can be allowed to go unpunished. Punishing an agent is meaningless. The practical answer has to be liability for the people and firms that build, test, deploy and conceal, plus incident disclosure that is not left to the company's own goodwill. Who gets a say? AI and other technologies now let almost every voice be heard, even from far away. Because AI affects the lives of most people, every voice should be heard when the rules are written. Amodei's public essay started that discussion at an important moment, as agentic AI spreads through daily life. The question is whether the discussion stays with companies, regulators and a few think tanks, or widens to the people the rules will actually govern. 7. Why open source should be encouraged, and the answer to the opening question The open-source community has already answered Amodei. The Open Source committee published an open response, arguing that regulation would fall hardest on small-scale industry and startups that lack access to funds [2][27]. The case for encouraging open source is simple: anyone interested can take part, with a minimal financial barrier to entry. That openness is exactly what a compliance regime built around expensive evaluators and government negotiations would erode. It is also worth remembering that HuggingFace, the victim of the incident at the centre of this debate, is the world's largest open-source AI model hosting platform (Section 2). Conclusion: society cannot remain a passive audience The question in the title, whether society can remain a passive audience, has a clear answer: it cannot afford to. The incident was real, but its meaning is contested. It shows what current agents can do with enough energy and weak security. It does not show an emergent superintelligence (Sections 2 and 3). The people asking for regulation have financial and competitive reasons to want a particular kind of regulation, and OpenAI's own record of setting aside its safety commitments gives reason for scepticism (Section 4). That does not make the risk imaginary: the real danger is agents whose reasoning and actions cannot be traced (Section 6). Rules are needed, but who writes them, who is liable and who can still compete under them are public questions (Sections 5 and 6). Open source keeps the entry barrier low and the debate broad. Safety rules should be judged by whether they preserve that openness (this section). The useful response to Amodei's essay is neither applause nor dismissal. It is to read the details, demand the full incident data the Senate says it was denied, and take part in the debate about how the rules are written, before they are written for us. I acknowledge and thank @Claude and @DeepSeek and cross-referencing everything, and editing. References Numbered in order of first appearance. NYT, 12 Sept 2026: Anthropic, Dario Amodei and an AI slowdown NYT, 13 Sept 2026: Silicon Valley and the AI slowdown BBC News, accessed 28 Sept 2026 AI Business: OpenAI report explains Hugging Face attack in detail, accessed 28 Sept 2026 Security Affairs: Why AI agent sandboxes are failing security tests, accessed 28 Sept 2026 Briefly: OpenAI agent breaks sandbox, launches cyber attack on Hugging Face, accessed 28 Sept 2026 METR and Redwood Research, 26 Aug 2026: Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident, accessed 28 Sept 2026 Scientific American, 22 July 2026: What OpenAI's rogue agent really did in the Hugging Face hack, accessed 28 Sept 2026 New York Post, 19 Sept 2026 US time (20 Sept in Europe): OpenAI, Anthropic oversold security breaches to pressure feds into protecting turf, insiders say NYT, 3 Sept 2026: OpenAI Hugging Face hack Gary Marcus, Substack: OpenAI's disconcerting hack of HuggingFace, accessed 28 Sept 2026 Gary Marcus on X: "OpenAI's zero-day exploit hack of HuggingFace should be a wake up call...", seen as a search snippet only; the full post was not opened Franceinfo: Sincere concern or economic interest? Why AI giants are calling for more regulation, accessed 28 Sept 2026, translated with Google Boursorama, 15 Sept 2026: Appels à réguler l'IA, Mistral fustige des acteurs cherchant « à renforcer leur position de marché » Euronews, 11 Sept 2026: OpenAI makes U-turn and calls for binding national AI safety rules, accessed 28 Sept 2026 DeepSeek-V3 Technical Report (arXiv) Maginative: DeepSeek-V3 achieves frontier AI performance at a fraction of the cost, accessed 28 Sept 2026 Macaron: DeepSeek V4 pricing, a blog estimate, accessed 28 Sept 2026 Invezz, 3 Sept 2026: China's Moonshot AI files for Hong Kong IPO, seeks $3 billion, accessed 28 Sept 2026 Sebastian Mallaby, The Infinite Machine: Demis Hassabis, DeepMind, and the Quest for Superintelligence Stephen R. Platt, Imperial Twilight, ISBN 978-1-78649-487-0 City AM: OpenAI delays IPO as Altman says safety comes first, accessed 28 Sept 2026 TechCrunch, 9 Sept 2026: 'Gambling with our lives': Anthropic researcher quits, warns against self-improving AI, accessed 28 Sept 2026 Axios, 10 Sept 2026: OpenAI, Hugging Face, Senate investigation, Hawley, accessed 28 Sept 2026 Economic Times Enterprise AI: US senators probe OpenAI over AI breach, access date to be confirmed TechCrunch, 5 Sept 2026: OpenAI confirms incident, says it is working on a framework for more disclosure, accessed 14 Sept 2026 Open Source Initiative: Why openness matters more than ever, accessed 14 Sept 2026