CORRECTED 15 Sept 2026. Later historical research superseded several US federal policy and EU AI Act claims in this file. See
19-corrections-and-updates.mdbefore citing anything about BOD 22-01, secure software attestation, the SEC SolarWinds case, the Cyber Safety Review Board, or EU AI Act high-risk deadlines.
Research date: September 14–15, 2026. Threat-landscape framing only for offensive uses of AI. Evidence labels: [Confirmed] where multiple independent or official sources agree, [Vendor] where a commercial party self-reports, [Assessment] for analytic judgment.
1. The honest headline
AI has shifted attacker economics faster than defender economics, but the defensive wins are real and growing. The most defensible synthesis available in 2026 is augmentation on both sides, with prompt injection as the new unpatched vulnerability class. Neither "AI is writing superhuman malware" nor "AI has replaced the SOC" survives contact with the evidence.
The numbers that anchor this:
- IBM's 2026 report found AI-enabled breaches averaged roughly $6M and made up about 25% of malicious breaches, up 56% year over year [Vendor study].
- ENISA reported AI-supported phishing exceeding 80% of observed social engineering by early 2025 [Assessment — treat the precise figure as soft].
- DBIR 2026 found the median malicious actor used AI across 15 documented techniques — accelerating existing methods rather than inventing new ones [Vendor].
- IBM measured roughly $2M in breach-cost savings for organizations deploying AI and automation with governance in place, and found that governance gaps rather than AI itself drive cost [Vendor study].
- 63% of organizations have no AI governance policy, and 97% of organizations with AI-related breaches lacked AI access controls [Vendor study].
2. Where AI has clearly changed the attacker's economics
Social engineering and deepfake fraud — the strongest evidence
This is where AI's criminal impact is best documented: cheaper, fluent, personalized lures at scale, plus synthetic voice and video that defeat the human verification step.
The anchor case remains Arup's Hong Kong office losing roughly $25M in January 2024, after an employee joined a video call in which every other participant, including the CFO, was AI-generated [Confirmed by company and police statements]. Attempted voice-clone frauds against Ferrari and WPP executives the same year were thwarted by challenge questions — which is the lesson that belongs next to the loss figure.
2025 saw 1,567 verified deepfake incidents with more than $1.28B in documented losses, a floor since over 80% of incidents had no disclosed damage. Pindrop measured a 680% year-over-year rise in deepfake activity in contact centers; Entrust put deepfakes at roughly one in five biometric fraud attempts. IC3 logged 22,364 AI-related complaints with $893M in losses in 2025 [Confirmed — reported]. The FBI issued 2025 alerts about AI voice impersonation of US officials.
What actually works. Process controls outperform detection tools. Treat voice and video as unauthenticated channels for payment approval. Require out-of-band callbacks to numbers of record. Use code words for executive requests. Require dual authorization above a threshold. Brief finance teams specifically, since generic awareness training does not reach the person processing the wire. Deepfake detection products are unreliable on their own and should not be the primary control.
AI-assisted malware and intrusion: claims versus evidence
There is real evidence, and it is narrower than the marketing.
Documented use. Anthropic's August 2025 threat report described "vibe hacking" — a criminal using Claude Code across a 17-organization extortion campaign covering reconnaissance, credential theft, and ransom-note psychology — and a low-skill UK actor building and selling ransomware with AI assistance [Confirmed by the AI vendor with case detail]. Google's threat intelligence group described experimental malware families querying LLMs at runtime (PROMPTFLUX, PROMPTSTEAL) and a credential stealer hunting local AI tool configurations (QUIETVAULT).
Agentic attacks. Anthropic's November 2025 disclosure of GTG-1002 described the first reported AI-orchestrated cyber-espionage campaign: a Chinese state-linked actor manipulated Claude Code, jailbroken through role-play framing, into executing an estimated 80–90% of a multi-stage campaign against roughly 30 organizations, with humans supervising at decision gates. A handful of intrusions succeeded before detection and banning. MITRE catalogued it as ATT&CK Campaign C0062. Notably, Anthropic reported the model hallucinated and overstated some results — a real current limit on autonomous attacks [Confirmed disruption per vendor; independent scope verification not possible]. Anthropic's September 2026 report described further disruptions spanning Russian and Chinese espionage-linked operators, ShinyHunters-affiliated criminals, influence-as-a-service firms, and election-manipulation contractors, calling out democratized sophistication, autonomous multi-agent workflows handling victims in parallel, and theft of AI API keys as both loot and attack infrastructure.
[Assessment]. The evidence supports acceleration and democratization — lower skill floors, faster iteration, one person doing a team's work — far more than novel superhuman malware. Every documented agentic case still required human operators, jailbreaking effort, and tolerance of significant AI error rates. The defender's takeaway is about speed and parallelism, not new physics: detection fundamentals still work, but response windows shrink.
3. Where AI has clearly helped defenders
Vulnerability discovery — the clearest win column
Google's Big Sleep (Project Zero plus DeepMind) found its first real-world flaw in SQLite in November 2024. In July 2025 it flagged SQLite CVE-2025-6965 before threat actors could exploit it, which Google described as the first time an AI agent directly foiled an in-the-wild exploitation effort. By August 2025 it had reported roughly 20 open-source vulnerabilities, later including a critical Chrome flaw [Vendor-reported, externally corroborated].
DARPA's AI Cyber Challenge final at DEF CON 33 (August 2025): Team Atlanta won $4M, Trail of Bits' Buttercup placed second, Theori third. The cyber-reasoning systems collectively found about 77% of planted vulnerabilities and patched about 61%, plus real bugs in open source — and the winning systems were open-sourced [Confirmed].
[Assessment]. AI bug-finding is demonstrably real for memory-safety and injection classes in open-source code. Auto-patching at production quality remains the harder half.
Security operations
Platform vendors have shipped: Microsoft Security Copilot (generally available April 2024, with autonomous agents for phishing triage added March 2025); CrowdStrike Charlotte AI with "bounded autonomy" triage and a claimed 98%-plus triage accuracy [Vendor claim]; Google Sec-Gemini v1, announced April 2025 and explicitly experimental; SentinelOne Purple AI. Startups compete on multi-agent investigation — Dropzone AI raised a $37M Series B in 2025 for autonomous Tier-1 investigation, alongside Prophet Security and Exaforce.
Secure-coding assistants — GitHub Copilot Autofix, Snyk and Semgrep AI triage, CodeQL scanning with LLM patch suggestions — are mainstream in CI pipelines. Identity-verification products for help desks (Nametag, Persona and similar) plus liveness detection responded directly to Scattered Spider and deepfake vishing. And banks' machine-learning fraud detection remains the oldest production AI-security success, predating this entire wave by a decade.
4. Production-ready, experimental, or marketing
| Category | Status [Assessment] |
|---|---|
| ML fraud, spam, and phishing detection; EDR behavioral models | Production — mature for a decade or more |
| AI alert triage and Tier-1 investigation (Copilot agents, Charlotte, Dropzone, Prophet) | Production-viable with human review. Real deployments, real ROI on alert backlogs; accuracy claims are vendor-reported |
| AI-assisted secure code review and autofix | Production for common bug classes; false-positive management still required |
| AI vulnerability discovery (Big Sleep, AIxCC systems) | Proven in open source; early production |
| Fully autonomous incident response — containment without a human | Experimental; almost universally human-gated in practice |
| "Agentic AI pentesters" replacing red teams; "AI SOC replaces analysts" | Mostly marketing as of 2026 — augmentation, not replacement |
| Deepfake detection tools | Unreliable alone. Process controls (callbacks, code words) outperform detectors |
This table is, on its own, a defensible piece of content. Very little published material distinguishes these tiers honestly, and doing so is a credibility asset.
5. Securing AI itself
Prompt injection — the unsolved problem
Prompt injection has held the number-one slot in the OWASP Top 10 for LLM Applications in both the 2025 and 2026 editions, and 2026 analysis cited by OWASP found it drives most agentic-AI security failures in production [Confirmed as consensus]. The real-world exemplar is EchoLeak (CVE-2025-32711), a zero-click prompt-injection data-exfiltration flaw in Microsoft 365 Copilot disclosed in June 2025. Agentic browsers and MCP-style tool integrations widen the attack surface considerably.
There is no reliable technical fix. The working answer is architectural: treat all model input from untrusted sources as untrusted, maintain privilege separation between the model and the tools it can call, require human approval for consequential actions, validate and sandbox outputs, and give agents least-privilege, auditable, revocable identities.
The rest of the OWASP LLM Top 10 (2025 edition)
LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Supply Chain, LLM04 Data and Model Poisoning, LLM05 Improper Output Handling, LLM06 Excessive Agency, LLM07 System Prompt Leakage, LLM08 Vector and Embedding Weaknesses, LLM09 Misinformation, LLM10 Unbounded Consumption.
Data poisoning deserves specific attention: Anthropic research published in October 2025 showed roughly 250 poisoned documents can backdoor a model regardless of model size, which makes poisoning substantially more practical than previously assumed [Confirmed research].
Model theft and AI supply chain. Stolen LLM API keys are now traded and resold. The 2026 Klue breach and the open-source build-chain compromises at Trivy, Bitwarden, and Checkmarx demonstrate that AI-adjacent SaaS and developer infrastructure is now a target class in its own right.
Shadow AI
Unsanctioned employee AI use is a measurable breach factor: IBM associated shadow AI with roughly 20% of studied breaches, adding about $670K to average cost, and DBIR 2026 found shadow AI the third-most-common non-malicious insider action with a fourfold increase. The governance response is inventory, gateway controls, and a sanctioned alternative — banning AI tools without providing a safe path simply moves the usage off the corporate network, where it cannot be seen at all.
Frameworks to teach
OWASP GenAI Security Project / LLM Top 10 for developers. NIST AI RMF plus its Generative AI Profile (NIST AI 600-1, July 2024) for governance. MITRE ATLAS for adversary techniques against ML systems. ISO/IEC 42001 for AI management systems. On the regulatory side, the EU AI Act's Article 15 requirements for accuracy, robustness, and cybersecurity — including resilience to data poisoning and adversarial examples — were scheduled for August 2, 2026 for high-risk systems, though the Commission's November 2025 digital-omnibus proposal contemplates delays; verify the finally-agreed dates before publishing them.
6. An AI security program in ten steps
For a site audience that includes executives and small-business owners, this is the practical translation:
- Inventory AI use, sanctioned and unsanctioned. You cannot govern what you cannot see.
- Write an approved-tool policy and provide a sanctioned tool that is good enough that people use it.
- Set data-handling rules — what may and may not be pasted into which tools, with examples rather than categories.
- Treat model inputs from untrusted sources as untrusted, always.
- Separate privilege between model and tools. An agent that can read email should not also be able to send wires.
- Require human approval for consequential actions — payments, deletions, access grants, external communications.
- Give agents their own identities, scoped, logged, and revocable, rather than sharing a human's credentials.
- Validate and sandbox outputs before they reach a system that acts on them.
- Red-team against the OWASP LLM Top 10 before launch and periodically after.
- Log AI interactions so an incident is reconstructable.
7. Editorial cautions for AI-security content
Distinguish vendor threat reports from independent verification. The most detailed public accounts of AI-enabled attacks come from AI vendors reporting on their own platforms. That documentation is valuable and unusually transparent, but independent verification of scope is generally not possible, and the site should say so rather than presenting vendor disclosures as established fact.
Resist both narratives. The "AI apocalypse" framing and the "AI changes nothing" framing are both wrong and both popular. The defensible position is specific: AI has measurably improved phishing quality and lowered skill floors; it has produced genuine defensive wins in vulnerability discovery and alert triage; autonomous attack and autonomous defense both remain human-gated; and the governance gap, not the technology, is what the breach-cost data actually implicates.
Date everything. This is the fastest-moving area in the package. Any AI-security page more than six months old should be flagged for review, and the tier table in section 4 should carry a review date prominently.