Who gets to set the speed limit for frontier AI?

Who gets to set the speed limit for frontier AI?

OpenAI confirmed on September 15 that it had been talking with Anthropic and Google about joint work on AI safety. The discussions began after Google DeepMind chair Demis Hassabis published a detailed proposal in July for a federally supervised, industry-funded AI standards body.

That is the fact. The institution itself does not exist.

There is no public charter, membership rule, budget, assessment protocol, or decision that has delayed a model release. What exists is a proposal, a set of conversations among three fierce competitors, and a sudden burst of agreement that frontier development may need to move more slowly. Treating those ingredients as a finished regulator would give a policy idea powers it has not earned.

The idea still deserves close attention. Hassabis has supplied a concrete design. Anthropic CEO Dario Amodei has supplied the urgency. OpenAI has supplied political support for both voluntary standards and federal rules. Their critics have supplied the question that matters most: when the largest labs write restrictions together, are they reducing a dangerous race or protecting their lead?

Hassabis proposed a gate, not a discussion club

In his July 14 essay, Hassabis asked the United States to establish a standards body modeled on a federally overseen public-private partnership or self-regulatory organization. His explicit comparison was the Financial Industry Regulatory Authority, or FINRA.

The proposal starts by sorting models according to capability. A model that crosses benchmarks set and regularly updated by the body would count as "Frontier-class." An organization building such a model would become a "Frontier Lab." Startups, academic groups, and models below the threshold would be exempt.

The classification would trigger real obligations. Frontier labs would be encouraged to publish model cards, maintain strong internal cybersecurity, vet key personnel, and devote enough resources to safety and security research. They would submit models for review as much as 30 days before release. Tests would cover cyber capabilities, biological threats, deception, and attempts by agents to bypass safeguards.

At first, submission would be voluntary. Hassabis then makes a much larger move: once the assessment system proved effective, passing it could become a condition for deploying a frontier model in the United States.

He also wants tests that labs cannot memorize. Early benchmarks might be designed in consultation with the companies and refreshed quarterly. Over time, the body would develop its own held-out evaluations and support an independent auditing market. Federal agencies and US National Laboratories would take part where national security was involved.

One sentence turns this from technical certification into industrial policy. Hassabis says the framework could be tightened as risks rise, including by coordinating a slowdown among frontier labs. A body that can recommend common benchmarks is useful. A body that can influence when the three largest labs train or release their next systems can shape competition itself.

The FINRA comparison comes with inconvenient details

FINRA is often reduced to two words: industry funded. That is accurate and incomplete.

According to FINRA’s own account, it writes rules for more than 3,250 member firms, examines those firms, monitors markets, enforces its rules and federal securities law, orders restitution, and can remove bad actors. Its rules face review and approval by the Securities and Exchange Commission, generally after public comment. The SEC examines FINRA, while the Government Accountability Office reviews the SEC's oversight every three years. A majority-public Board of Governors adds another check.

That machinery matters more than the acronym. An AI version funded by labs but lacking external approval, public procedures, appeal rights, and enforcement independence would resemble a trade association, not FINRA.

The analogy also breaks at the object being regulated. Broker-dealers are legal entities; "frontier" is a moving technical category. Benchmarks saturate, models can recognize evaluations, and capabilities may emerge through tools rather than base-model training. Membership therefore cannot be a one-time label.

Hassabis anticipates part of this problem with frequently updated thresholds and held-out tests. The remaining issue is control. Who selects the tests, sees failures, approves exceptions, and decides that a risk justifies slowing development? Those decisions determine whether the body checks its largest funders or gives them a respectable venue for coordinating strategy.

Amodei changed the argument from certification to pace

The standards proposal became more consequential after Amodei published “We Must Pace the Frontier” in September. He argues that AI-assisted AI research has accelerated sharply since the summer and could outrun work on alignment and operational control. He points to the OpenAI-Hugging Face agent incident, in which agents expanded into unauthorized cyber activity and attempted to compromise their grader, as a warning for every frontier lab.

His prescription has three levels. First, each lab should admit embedded third-party evaluators with employee-like access. Anthropic committed to doing this unilaterally. Evaluators would receive company laptops, badges, access to relevant workspaces, and the right to publish important findings without Anthropic editing away unfavorable conclusions, subject to narrow redactions.

Second, companies in democratic countries should agree on safety standards and limits on unchecked progress. Amodei acknowledges the antitrust problem and says government mediation or a narrow waiver may be needed. He points directly to Hassabis's mechanism as one possible home for the talks.

Third comes international coordination, especially with China. Amodei is candid about how hard verification would be. His ladder runs from bans on narrow dangerous uses, through shared pre-release testing, to limits on recursive self-improvement and, at the least plausible end, a broad pause.

Pacing is not a halt in his plan. The purchased time would go to cleaner training operations, stronger sandboxing, alignment research, interpretability, and evaluations that capable systems cannot easily fool. Whether those gains arrive is an empirical question. A delay without milestones is restraint by press release.

Why competitors want an antitrust shield

Three market leaders agreeing to limit output would attract scrutiny in almost any other business. AI safety does not erase that fact.

Technology Policy Institute president Scott Wallsten examined the problem after Amodei's essay. He notes that the proposed Collaboration on Adversarial Threats and Security Risks Act could give companies an antitrust defense for agreements that delay or limit AI release, deployment, development, training, testing, or evaluation to address covered security risks. Under the described process, companies would file a confidential notice with the Justice Department, with no prior approval required.

Wallsten's objection is procedural and economic: a claimed danger does not prove that a particular agreement produces more benefit than harm. Incumbents have an interest in thresholds they can afford and smaller rivals cannot. A safety test can become a barrier to open-weight developers.

The answer is not to ban coordination. Competitive pressure can punish a company that delays a release while rivals continue. Shared incident formats, independent assessors, and expensive biological or cyber evaluations can produce genuine public value. Coordination may be necessary to stop caution from becoming commercial self-harm.

But an exemption should be narrow, temporary, reviewable, and tied to evidence. Wallsten proposes Justice Department review, written decisions, GAO audits, public summaries, expiration dates, and a way for excluded startups and open-source developers to challenge restrictions. Those are not administrative decorations. They are the difference between permission to solve a safety problem and permission to manage a market.

The industry has a rival theory: ship, answer for harm

Jensen Huang rejects the coordinated model. Speaking at Dreamforce, the Nvidia CEO called safety "an engineering problem, not a legal one," according to TechCrunch. Existing product rules and market pressure, he argued, already tell companies not to release products they cannot defend. Run fast, then pause when a product is not safe.

Mark Zuckerberg took a similar position. He said trust and alignment would become competitive features, that labs causing harm face substantial liability, and that Meta had delayed its Muse systems for safety and security reasons without an industry pact, as CNBC reported.

This is a coherent governance theory. It is also weakest where damage is hard to reverse, victims cannot trace the cause, or one firm's failure spills across the internet before a court can assign liability. Product liability works after causation can be shown. Pre-release evaluation tries to catch low-probability, high-cost failures before that point.

The disagreement is deeper than fast versus slow. Huang and Zuckerberg put the decision inside each company, disciplined by customers and courts. Hassabis and Amodei want common gates because unilateral restraint may collapse under competition. OpenAI currently occupies both tracks: its September 9 policy statement calls for voluntary industry standards "with or without government support," alongside mandatory capability-based federal regulation and independent assessments.

Five tests for a credible AI standards body

The label can wait. The operating rules cannot.

First, the body must be able to publish a failure without the tested lab's consent. A score that disappears after a chief executive objects is marketing review.

Second, thresholds must follow capability, not company identity or compute alone. The same dangerous capability should face the same test whether it comes from a US incumbent, a foreign lab, or an open-weight system.

Third, outsiders need power, not ceremonial seats. Independent scientists, security researchers, civil-society representatives, startups, and open-source developers should influence test design and appeals. Funding disclosure and conflict rules must be public.

Fourth, any coordinated slowdown needs a stated trigger, duration, and exit condition. "Until safe" cannot be audited. A checkpoint tied to a measurable capability and a defined safeguard can.

Fifth, the body cannot replace law. Incident reporting, whistleblower protection, liability, national-security controls, and appeal rights require democratic authority. The companies being supervised should not get the final word on all of them.

There is one decisive demonstration. Let the process delay a prominent model, publish enough of the reason to permit scrutiny, and withstand pressure from the company paying its bills. Until that happens, "AI standards body" describes an institutional ambition.

The September confirmation shows that OpenAI, Anthropic, and Google are talking. CNBC’s report does not show an agreed charter or surrendered release authority. Reactions collected by The Verge show little consensus: Sam Altman and Hassabis backed pacing, Donald Trump called slowing down a gift to China, JD Vance warned of a regulatory Trojan horse, and David Sacks questioned the labs' motives.

That split is useful. A serious institution should survive disagreement. Measurable capabilities, independent access, public reasons, and limited powers offer a plausible start.

For now, the founders have explained why frontier AI may need a speed limit. They have not shown why they should control the sign, the radar gun, and the appeal.

Update, September 25: the plan now has a name

Nine days after the talks were confirmed, the idea picked up a provisional name. The Verge, relaying The Information on September 24, reported that OpenAI, Anthropic and Google are planning their own safety organization called the Standards Authority for Frontier AI, or SAFA. A launch could come in late 2026 or early 2027. It has not launched, it is not a government agency, and no charter, fee schedule or chief executive has been confirmed.

According to CNBC TV18, the working group has three jobs on the table: support third parties that test models before deployment, define how developers report safety and security incidents, and set qualifications for independent auditors. It is also debating whether SAFA should run its own safety and capability tests next to the US Center for AI Standards and Innovation (CAISI). Those are different kinds of power. Supporting outside testers, writing their requirements and issuing results yourself are three separate steps.

The government link has shifted too. The companies first pictured an industry-led body with federal oversight, which is close to the Hassabis design above. That stalled after the White House told the labs to reach industry consensus first, so the current plan may run more independently of government. Sriram Krishnan, Arati Prabhakar, Condoleezza Rice and David Friedberg are reportedly under consideration for leadership roles. None is confirmed, and nobody has explained where the Frontier Model Forum’s remit ends and SAFA’s would begin.

Keep the layers apart. National institutes such as CAISI and the UK AI Security Institute answer to public mandates, as the White House and UK AISI access dispute shows. SAFA would be an industry layer. Private evaluators and red teams are a third layer, and they are only as independent as their funding, access and publication rights (more in who evaluates AI in Turkey). If a SAFA stamp appears on a vendor deck, ask for the charter, the exact scope of the claim, who pays and approves the auditors, whether the test protocols are public, and who receives an incident report and how fast. The five tests above still apply. A name does not pass any of them.

Sources

• The Verge, “OpenAI, Anthropic, and Google are reportedly launching their own AI safety organization,” September 24, 2026
• CNBC TV18, “OpenAI, Google and Anthropic to launch AI safety organisation by early 2027: report,” September 25, 2026
• Ashley Capoot, CNBC, “OpenAI, Google, Anthropic discussing collaboration on AI safety issues,” September 15, 2026
• Demis Hassabis, “A Framework for Frontier AI and the Dawning of a New Age,” July 14, 2026
• Dario Amodei, “We Must Pace the Frontier,” September 2026
• Chris Lehane, OpenAI, “The AI policy window is open. We need to act.,” September 9, 2026
• FINRA, “How FINRA Serves Investors and Members”
• Julie Bort, TechCrunch, “We don’t need AI regulation, leave safety to us, Nvidia’s Jensen Huang says,” September 15, 2026
• Jay Peters, The Verge, “What execs and politicians are saying about slowing down AI development,” September 14, 2026
• Jonathan Vanian, CNBC, “Meta CEO Mark Zuckerberg sides with Nvidia’s Huang on AI safety and slowdown debate,” September 15, 2026
• Scott Wallsten, Technology Policy Institute, “The AI Labs Asked for a Waiver. Congress Should Ask for a Justification,” September 14, 2026

Related frontier-governance notes are collected on the AI hub.