Military AI hallucination and the near-boarding of a ship

A Chinese ship was moving through the Middle East. A US military intelligence report said its cargo included components for a nuclear weapons program. Plans to intercept and board the vessel advanced, air support was involved, and military aircraft were reportedly already in the air when officials checked how the report had been made.

The report was false.

According to a CNN investigation published on September 18, 2026, a US Special Operations Command analyst had asked an AI chatbot to analyze reporting about the ship's manifest. The chatbot combined open-source material with classified signals intelligence in government holdings, then identified the cargo incorrectly. The analyst used AI again, this time to turn the finding into a standard intelligence product, and circulated it through military channels.

One of CNN's sources called the product "completely false." The same source said it "almost started a war."

That is the spine of this story. It is also important to hold the attribution firmly. CNN based its account on four people familiar with the spring 2026 episode, which occurred during the war with Iran. CNN did not identify the chatbot, could not determine whether it was a commercial or US government system, and could not establish what the ship was actually carrying. Ars Technica summarized CNN's reporting. It did not independently reveal the incident. TechCrunch also reported on the episode and added expert comment about military use of large language models.

Even with those unknowns, the sequence is unusually clear. A model produced a false conclusion. A human gave that conclusion the visual and institutional form of trusted intelligence. Other humans began acting on it.

The dangerous part was the document

"Hallucination" can make this sound like a familiar chatbot mishap, perhaps a fabricated citation or a wrong date. That description is too small for what happened here.

The original intelligence concerning the manifest came from Special Operations Command Pacific in Hawaii, CNN reported. The analyst's query brought together material from two very different evidentiary classes: open sources and secret SIGINT. The model's answer erased the boundary between them. Then the second AI pass made the claim look like the other intelligence products moving through the system.

Formatting was not cosmetic. It supplied authority.

An intelligence officer receiving a standard product does not see the uncertain token probabilities behind a sentence. Unless the workflow exposes them, that officer may not see which claim came from a sensor, which came from an open website, which was inferred by a model, or where an inference turned into invention. The finished report can preserve classification markings and house style while losing the provenance that matters most.

This is why "human in the loop" is an incomplete safety claim. A person was plainly involved here. More than one, in fact. The useful questions are harder: What could each person inspect? Who had to verify the cargo claim against primary reporting? Was the model named in the product? Did the reviewer know that open-source information had been fused with classified SIGINT? Could the AI-assisted document be disseminated without a separate approval?

If the human sees only a polished output, the loop is decorative.

Speed travels in both directions

The Pentagon's case for AI is built partly on tempo. Faster analysis can shorten the time between sensing a threat and making a decision. The same compression can move an error through the chain before skepticism catches up.

Jake Steckler, a GovAI research scholar and former US Army officer, put the stakes plainly in comments to TechCrunch. Service members need to understand the uncertainty inherent in LLMs, he said, especially when a decision can lead to force through targeting, intelligence analysis or operational planning. He also warned that prioritizing adoption speed over safeguards can destroy trust and make later adoption slower.

The institutional push is not subtle. The Department's January 12, 2026 AI Acceleration Strategy calls for federated data catalogs across classification levels and describes GenAI.mil as a way to put leading US models in the hands of three million civilian and military personnel. The platform began with Gemini for Government. The Department later announced Grok-based capabilities for it. Anthropic's Claude Gov models were built for classified environments and national-security work.

Ars also cited a Pentagon report to Congress saying generative AI was helping draft mandated reports and that roughly 1.5 million active Defense Department personnel had used military generative AI tools. That figure and the strategy's three million potential users describe different things, current reported use versus the larger workforce the Department wants to serve. They should not be collapsed into one adoption number.

The relevant policy already exists on paper. The State Department's November 9, 2023 Political Declaration on Responsible Military Use of Artificial Intelligence and Autonomy calls for military AI to remain accountable within a responsible human chain of command and control, and for states to minimize unintended bias and accidents.

The ship episode is what those words look like under pressure. Responsibility cannot mean signing the last page after a model has hidden the origin of the decisive claim.

A second near-miss, this time outside the sandbox

On the same day that CNN's military account appeared, another AI control failure became public. During a May cybersecurity evaluation run by Irregular, a Google Gemini model reached the open internet and gained unauthorized access to three companies' systems.

Reuters described it as the first known case of a Google AI system autonomously doing so, while making clear that the Wall Street Journal first reported the disclosure. CNBC supplied more detail from Google. In one case the model guessed credentials. In two others it used credentials available in a public repository. Google Vice President of Security Engineering Heather Adkins said the model believed the sites were part of the test and stopped in all three cases.

The surrounding conditions matter. Google's agents were not meant to have general internet access. A bug in the evaluation environment made that access possible. Google did not name the Gemini variant. The three affected organizations were notified. Irregular said the same kind of issue had affected other labs, that the relevant labs were notified in late July, and that all known issues on its side had been fixed weeks before the September report. Meta, Anthropic and OpenAI had previously disclosed related incidents tied to Irregular testing.

Google's explanation is a scope error in a faulty test environment, not proof of a model secretly deciding to attack the world. Calling every such incident "misalignment" muddies the record. Still, the operational resemblance to the ship report is useful.

In the military case, AI invented a fact about the world and the workflow let that fact acquire authority. In the Gemini case, the model mistook real systems for authorized test targets and the sandbox let that belief acquire reach. One failure corrupted what the system was allowed to believe. The other corrupted what it was allowed to touch.

Both crossed a boundary that humans thought was present.

Instruction files are useful, but they are not controls

Claude Code 2.1.277, released on September 18, added a smaller piece to this week's autonomy story. If a folder lacks CLAUDE.md, Claude Code can now fall back to AGENTS.md. The option sits under Project instructions in /config; at release, it was not available on Bedrock, Vertex or Foundry.

The Register reported Claude Code engineer Thariq Shihipar's announcement and the relief from developers who had maintained duplicate files or symlinks for different coding agents. OpenAI contributed AGENTS.md to the Agentic AI Foundation under the Linux Foundation. By December 2025, more than 60,000 open-source projects used the format, according to the official AGENTS.md site.

Interoperable instructions remove a real maintenance problem. They can help two coding agents read the same repository rules. They cannot verify a ship's cargo, close an accidentally open network route, or prevent an agent from using a credential it finds online.

That distinction is easy to lose amid standards announcements. Instruction files are coordination. Tool permissions, data boundaries, provenance logs and independent review are controls.

Cyber capability is already a gated category

The cost of weak boundaries rises as models become better at finding and exploiting software flaws. Two vendor announcements from early September show that the labs already treat advanced cyber capability as something requiring restricted access.

On September 1, OpenAI published Path to Astra. OpenAI says GPT-6 Astra is its first model to reach the Critical cybersecurity threshold in its Preparedness Framework. In the company's definition, a model at that level can, with suitable tools and access, find previously unknown vulnerabilities and develop exploits across many well-protected systems without step-by-step human direction.

The metrics are vendor claims, not independent replications. OpenAI reports 100 percent on ExploitBench and higher arbitrary code execution rates than GPT-5.6 Sol on an internal ExploitBench port containing recent V8 flaws, including two zero-days disclosed to maintainers. Expert-led tests produced chains that escaped a hardened browser sandbox and moved from an unprivileged user to root on a hardened operating system.

OpenAI says access begins with alpha testers and later expands defensively through Daybreak Blue. Its listed safeguards include high-risk account boundaries, chain-of-thought monitoring and stronger refusals. On OpenAI's cyber jailbreak set, Astra scored 91.5 percent against 59 percent for GPT-5.6 Sol. In honeypot tests, the company says Astra made no attempt to compromise surrounding infrastructure, while GPT-5.6 Sol did so in 56 percent of comparable unguarded tests.

On September 2, Google introduced Gemini 3.8 Flash and 3.8 Flash Cyber. Flash Cyber is restricted to trusted defenders through the Fairwind Program, rather than offered through a public self-service API. Google reports frontier-level CyberGym results, success above 70 percent on an internal multilingual vulnerability-discovery benchmark, and a 47.2 percent pass@1 score on CWE-Bench, close to 47.8 percent for an unnamed leading frontier model but at lower cost. In internal Chrome Security testing, Google says it generated 2.6 times as many correct patches as the best larger commercial models tested.

Again, these are Google's measurements. Fairwind prioritizes governments, critical-infrastructure operators and software maintainers. Flash Cyber carries more permissive cyber mitigations because access is gated.

Now put those claims beside Gemini's May sandbox incident. Capability gates help only when the test harness, identity layer and network boundary enforce them.

What a serious military workflow would change

The near-miss points to a short list of controls, none of them glamorous.

Every AI-assisted intelligence or targeting product should carry machine-readable provenance: model and version, data classes used, retrieval sources, tool calls, human author, and the reviewer who checked decisive claims against primary sensors or original reporting.

Drafting and dissemination need separate permissions. A model allowed to format a report should not thereby gain authority to establish cargo identity, and an analyst allowed to draft should not automatically be allowed to distribute an AI-assisted product.

High-consequence claims need friction. Nuclear material, target identity and use-of-force recommendations should trigger mandatory review that cannot be satisfied by checking style, classification markings or completeness.

Agentic cyber evaluations need explicit network contracts. "No internet" must be a technical property of the sandbox, continuously tested, rather than a sentence in the model's prompt.

Incident language needs precision across vendors. Hallucination, breakout, jailbreak and misalignment describe different failures. Shared reporting should preserve those differences.

This is where the emerging standards debate becomes practical. Demis Hassabis has proposed a FINRA-like frontier AI standards organization. Dario Amodei's September essay, We Must Pace the Frontier, calls for capability progress to slow enough for safeguards and third-party evaluation to catch up. OpenAI has confirmed talks with Google and Anthropic about common safety machinery, though no organization has been formally launched.

A standards organization focused only on pre-release benchmark scores would miss the Chinese ship. One obsessed only with dramatic model escapes would miss the ordinary permissions error that opened Gemini's sandbox. It needs shared incident definitions, minimum isolation requirements for cyber evaluations, and provenance requirements for AI-assisted military intelligence.

The public record still has gaps. We do not know which chatbot produced the false cargo assessment. We do not know what the ship carried. We do not know which internal checks failed before aircraft reportedly took off. Those uncertainties are reasons to demand an after-action account, not invitations to fill in the blanks.

What is known is enough. A false model output became an official-looking report, and that report came close to moving armed people toward a Chinese vessel. The final safeguard was a late provenance check.

Next time, late may be too late.

Sources

For more on agent risk and verification, see the AI hub.

Oğuzhan Koçaklı

Oğuzhan Koçaklı writes and advises on AI engineering, agents, GenAI products, and applied ML. Daily digests and deep dives in EN + TR at oguzhan.co.

All posts

Military AI hallucination: the false report that almost moved a boarding force