The honest summary: AI has improved attacker economics faster than defender economics, the defensive wins are real but narrower than marketed, and prompt injection is an unsolved problem that nobody should be selling a solution to.
Here's the category-by-category version, because "AI security" now covers a dozen things with wildly different maturity levels.
The assessment table
| Category | Status |
|---|---|
| Machine-learning fraud, spam and phishing detection; behavioural endpoint models | Production — mature for a decade or more |
| AI alert triage and Tier-1 investigation | Production-viable with human review. Real deployments, real ROI on alert backlogs — but accuracy claims are vendor-reported |
| AI-assisted code review and autofix | Production for common bug classes. Still needs false-positive management |
| AI vulnerability discovery | Proven in open source, early production |
| Fully autonomous incident response — containment with no human | Experimental. Almost universally human-gated in practice |
| "Agentic AI pentesters" replacing red teams; "AI SOC replaces analysts" | Mostly marketing |
| Deepfake detection tools | Unreliable alone. Process controls outperform them |
If you take one thing from this page, take the bottom two rows.
Where AI genuinely helped defenders
Vulnerability discovery is the clearest win. Google's Big Sleep project found its first real-world flaw in SQLite in late 2024. In July 2025 it flagged another before attackers could exploit it — which Google described as the first time an AI agent directly foiled an in-the-wild exploitation attempt. By August it had reported around twenty open-source vulnerabilities.
DARPA's AI Cyber Challenge, concluded at DEF CON in August 2025, is better evidence still, because it was a controlled competition rather than a vendor demo. The competing systems collectively found about 77% of planted vulnerabilities and patched about 61%, plus real bugs in live open-source code. The winning systems were open-sourced.
The honest caveat: AI bug-finding is demonstrably real for memory-safety and injection classes in open source. Auto-patching at production quality remains the harder half.
Alert triage is the other real one. Tier-1 investigation is high-volume, pattern-heavy, and exactly what these systems do well. Deployments report genuine reduction in alert backlogs. IBM's 2026 breach-cost report measured roughly $2 million in breach-cost savings for organizations using AI and automation — but note the condition attached: with governance in place.
Where AI genuinely helped attackers
Social engineering, overwhelmingly. This is the best-evidenced criminal application: cheaper, fluent, personalised lures at scale, plus synthetic voice and video that defeat human verification.
The anchor case is Arup's Hong Kong office losing roughly $25 million in January 2024 — confirmed by the company and Hong Kong police, after an employee joined a video call where every other participant — including the CFO — was AI-generated.
But put the counter-example next to it: attempted voice-clone frauds against Ferrari and WPP executives the same year were stopped by challenge questions. The countermeasure is procedural, not technological — the same callback discipline covered in the scam centre and the ransomware playbook.
On malware, the evidence is narrower than the marketing. There are documented cases of criminals using AI assistants across extortion campaigns, and of low-skill actors building ransomware with AI help. In late 2025, Anthropic disclosed what it described as the first AI-orchestrated espionage campaign, with a model executing an estimated 80–90% of a multi-stage operation under human supervision.
Read that carefully, though. It required human operators, deliberate jailbreaking, and tolerance for significant error — the vendor noted the model hallucinated and overstated results. And that autonomy figure comes from a single source and drew scepticism from researchers.
The defensible read: AI is accelerating and democratising existing attack methods, not inventing new physics. DBIR 2026 found the median malicious actor used AI across 15 documented techniques — all of them techniques that already existed. One person can now do a team's work. That's a meaningful shift in speed and volume. It isn't a new category of threat.
Prompt injection: the part with no solution
Prompt injection has held the number-one spot in OWASP's Top 10 for LLM Applications in both the 2025 and 2026 editions, and 2026 analysis attributes most production failures in agentic AI systems to it.
Here's why it's hard, stated precisely: a language model's context window does not distinguish instructions from data at an architectural level. Everything is tokens. The system prompt, the user's message, and text retrieved from a web page all arrive through the same channel, and the model's compliance with instructions is learned behaviour rather than an enforced boundary.
Contrast SQL injection, which was solved — prepared statements structurally separate code from data. No equivalent structural separation exists for natural-language models.
The real-world demonstration was EchoLeak (CVE-2025-32711), a zero-click prompt-injection data-exfiltration flaw in Microsoft 365 Copilot disclosed in June 2025.
What actually helps is architectural, not a product:
- Treat all model input from untrusted sources as untrusted
- Separate privilege between the model and its tools. An agent that reads email should not also be able to send payments
- Require human approval for consequential actions
- Validate and sandbox outputs before anything acts on them
- Give agents their own scoped, logged, revocable identities
Be sceptical of anything sold as prompt-injection prevention. Mitigation, yes. Prevention, no.
Shadow AI is the measurable problem
While the industry debates autonomous attacks, the thing actually showing up in breach data is much more ordinary: employees pasting sensitive data into unsanctioned AI tools.
IBM associated shadow AI with roughly 20% of studied breaches, adding about $670,000 to average cost. DBIR 2026 found it the third most common non-malicious insider action, up fourfold.
And IBM's 2025 breach-cost study found 63% of organizations have no AI governance policy at all.
The response that works is not a ban — bans move the usage onto personal devices where you can't see it. It's inventory, a clear data-handling rule with examples, and a sanctioned tool good enough that people use it — an insider-risk problem, handled the way minimum viable security handles the rest.
A ten-step AI security programme
- Inventory AI use, sanctioned and not
- Write an approved-tool policy, and provide a tool people will actually use
- Set data-handling rules with concrete examples, not categories
- Treat untrusted model input as untrusted
- Separate privilege between model and tools
- Require human approval for consequential actions
- Give agents their own identities — scoped, logged, revocable
- Validate and sandbox outputs
- Red-team against the OWASP LLM Top 10 before launch and periodically
- Log AI interactions so an incident is reconstructable
IBM's central finding is worth repeating: it's the governance gap, not the technology, that the cost data implicates.
How to read AI security vendor claims
Ask what's measured, and against what. "98% triage accuracy" means nothing without knowing the alert population and who checked.
Ask whether a human is in the loop, and what happens when the model is wrong.
Ask about the failure mode, not the success rate. A triage system that closes real incidents as benign is worse than no triage system.
Treat AI vendors' own threat reports as valuable but unverifiable. The most detailed public accounts of AI-enabled attacks come from AI companies reporting on their own platforms. That transparency is genuinely useful. Independent verification of scope generally isn't possible, and we say so.
Discount anything claiming replacement. Augmentation is real and shipping. Replacement is a slide.
What to do next: Find out which AI tools your organization is actually using. Not which are approved — which are in use. That gap is where the measurable risk currently sits.
Related: Securing AI systems · The 2026 threat landscape · Minimum viable security · AI regulation tracker (planned reference; not yet available)
Sources: OWASP Top 10 for LLM Applications, 2025 and 2026 editions. IBM Cost of a Data Breach, 2025 and 2026 editions — AI-enabled breach costs, shadow AI, governance gap. Verizon 2026 DBIR — attacker AI technique use and shadow AI as insider action. Google — Big Sleep vulnerability discovery disclosures (2024–2025). DARPA — AI Cyber Challenge final results, DEF CON 33, Aug 2025. Anthropic threat intelligence reports (Aug 2025, Nov 2025, Sept 2026) — AI-assisted and agentic attack disclosures, treated as vendor-reported. Microsoft — EchoLeak (CVE-2025-32711). NIST AI Risk Management Framework and MITRE ATLAS.