1 hour ago
35
An American warplane was already airborne this spring, poised to intercept a Chinese cargo ship, when officials abruptly pulled back after realizing the intelligence behind the operation was fabricated by a chatbot. The episode, first reported by CNN and later detailed by TechCrunch and Ars Technica, has become one of the most alarming known cases of AI hallucination military intelligence failures to date — a near-miss that one source told CNN “almost started a war.”
Key takeaways
- A US intelligence report wrongly claimed a Chinese ship was carrying nuclear weapons program components, based on an AI chatbot’s hallucinated analysis.
- Military aircraft were already in the air, preparing an interception with air support, before officials discovered the error.
- The chatbot had fused open-source intelligence with classified signals intelligence, then formatted the false findings into an official-looking summary.
- The Department of Defense is simultaneously expanding its use of generative AI tools, including Google’s Gemini for Government and Grok for Government on its GenAI.mil platform.
- Anthropic’s Claude, also customized for US intelligence work, was blacklisted in March over the company’s opposition to using its models in autonomous weapons systems.
AI Hallucination Nearly Triggered US Military Interception of Chinese Ship
The false intelligence report originated with a US Special Operations Command analyst who used a chatbot to make sense of data tied to the Chinese vessel’s cargo manifest. According to CNN, “four sources familiar with the episode” said the resulting report claimed the ship was transporting components for a nuclear arms program through the Middle East, and that the intelligence circulated during the war with Iran. TechCrunch reported the tool was used twice: first to synthesize the material, and again to format the erroneous conclusion into a polished, official-looking summary that then moved up the chain of command.
False Identification of Nuclear Components
The chatbot, according to CNN’s reporting, “fused together open-source intelligence with secret signals intelligence in government holdings” — and got it wrong. The tool misidentified the ship’s actual cargo, producing a conclusion that had no basis in fact but read convincingly enough to be treated as credible intelligence.
Role of AI Chatbot in Generating Erroneous Report
This is where the danger of generative AI US military use becomes concrete: a single flawed output, once dressed up as a formal report, can travel through command channels without anyone questioning its origin. Jake Steckler, a research scholar at GovAI and a veteran US Army officer, told TechCrunch that “it’s important for service members to understand the uncertainty inherent to LLMs,” adding that this matters most “for any decisions that could lead to use of force, like targeting, intelligence analysis, or operational planning,” because “there are life and death consequences for those decisions.”
Potential for Military Conflict Averted
By the time officials caught the mistake, the US military was already preparing to intercept and board the ship with air support. The operation was called off at the last minute. One source described the near-disaster to CNN in blunt terms: it “almost started a war.” Ars Technica noted the case ranks among the more consequential known instances of a hallucinating AI system undermining a professional report — a phenomenon that has already tripped up authors, journalists, judges, doctors, police departments, and corporate call centers since “hallucinate” became Cambridge Dictionary’s word of the year in 2023. Researchers have suggested it may be effectively impossible to eliminate hallucinations from large language models altogether.
US Department of Defense’s AI Strategy and Tools
Despite the scare, the Pentagon’s push toward deeper military AI safety integration hasn’t slowed down — if anything, it’s accelerating. In January, the Department of Defense rolled out what it called an “AI acceleration strategy,” aimed at making “all appropriate data available across federated IT systems for AI exploitation, including mission systems across every service and component.”
The AI Acceleration Strategy Launched in January
Defense Secretary Pete Hegseth framed the initiative around data readiness rather than caution: “AI is only as good as the data that it receives, and we’re going to make sure that it’s there,” he said when unveiling the strategy. The goal, according to the department, is to speed up decision-making and preserve what the Pentagon has described as a critical edge in what it calls the “kill chain” — the sequence of steps from detecting a target to acting on it.
Use of Google’s Gemini and Grok on GenAI.mil Platform
Last December, the department announced it would build its bespoke “GenAI.mil” platform around Google’s Gemini for Government. Last month, it added Grok for Government as a second option on the same platform. By June, a Pentagon representative told Congress that generative AI tools were already being used to help draft congressionally mandated reports, and that 1.5 million active Department of Defense personnel had used the military’s generative AI tools in some capacity — a scale that underscores just how embedded these systems already are in day-to-day operations.
Anthropic’s Claude Model and Its Blacklisting
Anthropic also offers a customized version of its Claude model for US intelligence work. But the relationship between the company and the Pentagon has been anything but smooth. In March, the Department of Defense blacklisted Anthropic over the company’s refusal to allow its models to be used in autonomous weapons systems. A federal judge ruled last month that the blacklisting amounted to “unlawful retaliation in violation of the First Amendment” — a decision that highlights the friction between military demand for AI capability and the ethical limits some AI developers are trying to hold onto.
Regulatory and Ethical Considerations in Military AI Use
The near-miss lands squarely on top of long-standing warnings that AI defense risks require stronger human checks, not just faster systems. In 2023, the State Department issued a “Declaration on Responsible Military Use of Artificial Intelligence and Autonomy,” which stressed that “principled” use of AI by armed forces “should include careful consideration of risks and benefits” and should “minimize unintended bias and accidents.” The declaration also insisted that accountable use of AI must always involve “a human in the loop, a responsible human chain of command and control.”
State Department’s Principles on Responsible AI Use
That framework was meant to guide exactly the kind of scenario that played out this spring — one where an AI-generated report almost drove a real-world military action. Whether the human-in-the-loop principle was meaningfully applied before the interception order was given is precisely the question this episode raises, since the flawed conclusion made it far enough up the chain to trigger preparations for an armed operation.
Human Oversight and ‘Human in the Loop’ Mandate
In practice, the case shows how thin that human oversight layer can be when a chatbot’s output looks polished enough to pass as verified intelligence. This matters beyond one incident: as more of the Pentagon’s 1.5 million AI users lean on these tools for analysis, formatting, and reporting, the odds of a hallucinated conclusion slipping past review multiply, especially when speed is treated as the primary metric of success.
Rise of Autonomous Weapons and Ethical Concerns
The stakes are compounded by the broader trajectory of military AI. Fully autonomous attack drones have already been used in the Russian conflict in Ukraine and tested by NATO-backed military contractors, according to reporting cited by Ars Technica. Set against that backdrop, an AI hallucination that nearly triggered a naval interception looks less like an isolated glitch and more like an early warning sign about the pace at which autonomous and generative AI systems are being folded into decisions that carry life-or-death consequences.
FAQ
What caused the false intelligence report about the Chinese ship?
An AI chatbot hallucinated, incorrectly identifying the Chinese ship’s cargo as nuclear components by fusing open-source and secret intelligence.
How close did the US military come to acting on the false intelligence?
The US military was preparing to intercept and board the Chinese ship with air support before the error was discovered.
What AI tools does the US Department of Defense use for military AI operations?
The DoD uses Google’s Gemini for Government and Grok for Government within its GenAI.mil generative AI platform.
What guidelines exist for the responsible use of AI in the US military?
The State Department’s 2023 declaration emphasizes principled AI use involving human oversight and minimizing risks and bias.
Article produced with the assistance of artificial intelligence and reviewed by the editorial team.
English (US) ·