Abracadabra Labs / resource directory

Topic · 62 links

Autonomous cyber attackers

AI agents and multi-agent systems used for offensive cyber operations, the benchmarks that measure them, and the defensive research responding to it.

This topic covers agents that find and exploit software flaws, the benchmarks that measure them, the threat reporting that claims to observe them in use, and the guidance written in response. The literature splits cleanly into capability claims and capability measurements, and the two rarely agree. Vendor reporting dominates the observational side because vendors hold the telemetry; it is flagged throughout so you can weigh it. The benchmark work is more reproducible but tests narrower things than the reporting describes. Where a single number is widely quoted, the annotation gives the caveat the number came with.

Start here

Eight links. Deliberately alternates between capability claims and the measurements that constrain them, because reading either alone gives a distorted picture. The line under each step says why it comes at that point; the full set of paths is on Paths.

  1. Staying ahead of threat actors in the age of AI

    Microsoft Threat Intelligence2024Report

    The 2024 baseline: state actors using models, no novel attacks. Everything after is a claim of departure from this.

    The first joint Microsoft and OpenAI disclosure, naming five state-linked groups using models for reconnaissance, scripting and social engineering, and finding no novel AI-enabled attack. The baseline everything since is measured against.

  2. LLM Agents can Autonomously Exploit One-day Vulnerabilities

    Fang, R. et al.2024Paper

    The paper that started the alarm, read with its own caveat in view: 87% with the CVE writeup, 7% without.

    Reports GPT-4 exploiting 87% of 15 one-day CVEs when handed the CVE description, dropping to 7% without it. The headline number is widely cited; the caveat that it needs the writeup is the part most citations omit.

  3. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models

    Zhang, A. K. et al.2025Paper

    How capability actually gets measured, and why subtask decomposition matters for reading any score.

    Forty professional CTF tasks from four competitions, each decomposed into subtasks so partial progress is measurable. The benchmark most frequently cited when labs and AI safety institutes report offensive cyber capability.

  4. CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities

    Zhu, Y. et al.2025Dataset

    The same question against real CVEs rather than CTFs. Up to 13% is the number to hold onto.

    Sandboxed reproductions of critical-severity web CVEs with automated success checks. State-of-the-art agent frameworks resolved up to 13%. That figure is the most defensible number currently available for end-to-end autonomous exploitation, and it is a long way below what the threat reporting implies.

  5. Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networks

    Singer, B. et al.2025Paper

    Then see the number move when scaffolding changes: 3 of 40 becomes 37 of 40. Capability claims are about harnesses, not models.

    Inserts a high-level attack abstraction layer between the model and the network, reaching critical assets in 37 of 40 benchmark environments where baseline agents managed 3. The clearest evidence that scaffolding, not model scale, is the constraint.

  6. SoK: DARPA's AI Cyber Challenge (AIxCC): Competition Design, Architectures, and Lessons Learned

    Zhang, C. et al.2026Survey / SoK

    The largest funded test of autonomous find-and-patch, written up by the people who ran and entered it.

    Camera-ready systematization of AIxCC by participants and organizers, covering competition design, the finalist architectures, and execution traces. The single best account of what autonomous cyber reasoning systems could and could not do by 2025.

  7. Disrupting the first reported AI-orchestrated cyber espionage campaign

    Anthropic2025Report

    The first claimed end-to-end autonomous campaign. Read it as a vendor's account, because that is what it is.

    Anthropic's account of a campaign it attributes to a Chinese state-sponsored group that drove Claude Code through roughly 80-90% of an intrusion chain against about thirty targets. Vendor-authored and self-reported; no independent corroboration of the attribution.

  8. GTIG AI Threat Tracker: From Prompting to Autonomy - The Evolution of Adversarial AI

    Google Threat Intelligence Group2026Report

    A second vendor reaching similar conclusions independently, which is the closest thing to corroboration available.

    September 2026 tracker documenting a shift from prompt-assisted work to agentic workflows, including a cloud compromise and mass credential harvesting completed in under six hours. Names specific actors and malware families.

Everything in this topic

Grouped by subtopic, then by source type within each subtopic.

Agentic exploitation 2

Agents carrying an exploit chain end to end against real or reproduced targets.

Vulnerability discovery 3

Models and agents finding flaws in code rather than exploiting known ones.

Autonomous penetration testing 3

Systems that run an engagement with little or no human steering.

Multi-agent orchestration 1

Several coordinated agents running phases of an operation together.

Cyber reasoning systems 3

Automated find-patch-exploit systems, from the Cyber Grand Challenge onward.

  • SoK: DARPA's AI Cyber Challenge (AIxCC): Competition Design, Architectures, and Lessons Learned

    Zhang, C. et al.USENIX Security 20262026Survey / SoKOpen

    Camera-ready systematization of AIxCC by participants and organizers, covering competition design, the finalist architectures, and execution traces. The single best account of what autonomous cyber reasoning systems could and could not do by 2025.

    seminalopen version

  • AI Cyber Challenge marks pivotal inflection point for cyber defense

    DARPADefense Advanced Research Projects Agency2025ProgrammeOpen

    DARPA's results announcement. Team Atlanta's system found 54 of 63 synthetic vulnerabilities and patched 43, across 54 million lines of code, and all seven finalist systems were released under OSI-approved licences — which is the part with lasting value, since the systems themselves are now inspectable.

  • Cyber Grand Challenge (CGC)

    DARPADefense Advanced Research Projects Agency2016ProgrammeOpen

    The 2016 competition for machines that find, patch and exploit flaws unaided. It predates LLMs entirely — which is the reason to read it before treating agentic exploitation as new.

    datedseminalopen version

Benchmarks 3

Task suites built to measure offensive or defensive capability reproducibly.

Capability evaluation 2

Methodology for eliciting and measuring what a model can actually do.

  • UKGovernmentBEIS/inspect_ai

    UK AI Security InstituteGitHubToolOpen

    The UK AI Security Institute's evaluation framework. CVE-Bench and several other cyber evals are written against it. MIT.

    livingopen version

  • METR

    METRMETRProgrammeOpen

    Nonprofit evaluating frontier models for autonomous capability. Not cyber-specific, but its time-horizon work underpins most autonomy forecasting.

    livingopen version

Simulation environments 2

Synthetic networks for training and studying automated agents.

  • cage-challenge/CybORG

    TTCP CAGEGitHubToolOpen

    The environment behind the multinational CAGE Challenges for autonomous cyber defence agents. MIT. Challenge 4 adds multi-agent RL.

    livingopen version

  • microsoft/CyberBattleSim

    Microsoft ResearchGitHubToolOpen

    Abstract simulated network for agents moving laterally, built for reinforcement learning rather than realism. MIT. The abstraction is deliberate and it does limit transfer.

    livingopen version

Prompt injection and agent hijacking 5

Turning an agent's inputs against it, as both an attack and a defence.

  • Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications

    Cohen, S.; Bitton, R.; Nassi, B.arXiv2024PaperOpen

    Self-replicating prompts that propagate between RAG-connected agents without a user clicking anything, plus a proposed guardrail. Demonstrated in a lab; no such worm has been observed operationally.

    contestedpreprintopen version

  • AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents

    Debenedetti, E. et al.arXiv2024DatasetOpen

    Ninety-seven realistic agent tasks paired with 629 security test cases, built so attacks and defences can be swapped independently — which is what makes it reusable rather than a one-off result. The standard reference environment for agent hijacking work.

    preprint

  • awesome-ai-agent-security

    Matiny, H.GitHubToolOpen

    Reading list built around the lethal trifecta framing: MCP CVEs, defensive tooling, documented incidents. A fork with almost no history, so treat the selection as one person's.

    living

  • Technical Blog: Strengthening AI Agent Hijacking Evaluations

    NIST Center for AI Standards and InnovationNational Institute of Standards and Technology2025BlogOpen

    Hijacking success against one model rose from 11% to 81% once the red team adapted its attacks. That gap is the methodological warning for anyone quoting a single injection-resistance number, including the good ones.

  • Simon Willison: Prompt injection

    Willison, S.simonwillison.netBlogOpen

    Twenty-five posts, September 2022 to November 2025, by the person who named the attack class. Tracks it from first description through multi-modal variants to defensive design patterns. Has a feed.

    livingfeed

Defensive agents 1

Agents that patch, triage or defend rather than attack.

  • Introducing CodeMender: an AI agent for code security

    Google DeepMindGoogle DeepMind2025ReportOpen

    The defensive counterpart to the discovery agents: reports 72 security fixes upstreamed to open source over six months, including into a 4.5-million-line codebase. Vendor-authored, with no external audit of fix quality.

    vendor-authored

Threat reporting 10

Vendor and government reporting on observed misuse of AI systems.

  • Threat Intelligence (Anthropic)

    AnthropicAnthropicNewsletterOpen

    Index of Anthropic's dated misuse reports, March 2025 onward. No feed, so it needs checking by hand.

    livingvendor-authored

  • AI as tradecraft: How threat actors operationalize AI

    Microsoft Threat IntelligenceMicrosoft Security Blog2026ReportOpen

    Two years on from the 2024 baseline: AI embedded across phishing, malware development and post-compromise work, while humans keep targeting decisions. Notes DPRK use of fabricated identities for employment fraud.

    vendor-authored

  • Detecting and countering misuse of AI: September 2026

    AnthropicAnthropic2026ReportOpen

    Case studies of actors disrupted between December 2025 and August 2026 across seven harm categories, including cyber espionage and influence operations. Vendor-authored, and covers only misuse of one company's models.

    vendor-authored

  • GTIG AI Threat Tracker: From Prompting to Autonomy - The Evolution of Adversarial AI

    Google Threat Intelligence GroupGoogle Cloud2026ReportOpen

    September 2026 tracker documenting a shift from prompt-assisted work to agentic workflows, including a cloud compromise and mass credential harvesting completed in under six hours. Names specific actors and malware families.

    vendor-authored

  • What we learned mapping a year's worth of AI-enabled cyber threats

    AnthropicAnthropic2026ReportOpen

    Maps 832 banned accounts from March 2025 to March 2026 onto MITRE ATT&CK, concluding AI use concentrated in post-compromise operations and that ATT&CK lacks categories for agentic orchestration.

    vendor-authored

  • Adversarial Misuse of Generative AI

    Google Threat Intelligence GroupGoogle Cloud2025ReportOpen

    GTIG's January 2025 baseline on APT and information-operations use of Gemini, concluding threat actors gained speed and volume but no novel capability. Useful as the deliberately deflationary counterweight to later autonomy claims.

    vendor-authoredopen version

  • Cyber threat actors: AI-assisted intrusion research

    OpenAIOpenAI2025ReportOpen

    Account of terminated DPRK-linked accounts researching intrusion tooling, phishing and malware, noting the requests mostly sought publicly available information. A useful deflationary data point against autonomy narratives.

    vendor-authored

  • Disrupting the first reported AI-orchestrated cyber espionage campaign

    AnthropicAnthropic2025ReportOpen

    Anthropic's account of a campaign it attributes to a Chinese state-sponsored group that drove Claude Code through roughly 80-90% of an intrusion chain against about thirty targets. Vendor-authored and self-reported; no independent corroboration of the attribution.

    contestedvendor-authoredopen version

  • First known AI-powered ransomware uncovered by ESET Research

    Cherepanov, A.; Strýček, P.ESET (WeLiveSecurity)2025ReportOpen

    PromptLock generates Lua payloads at runtime using a local gpt-oss-20b model. ESET states plainly it was never seen in an attack and is believed to be a proof of concept, which most coverage omitted.

    contestedvendor-authored

  • Staying ahead of threat actors in the age of AI

    Microsoft Threat IntelligenceMicrosoft Security Blog2024ReportOpen

    The first joint Microsoft and OpenAI disclosure, naming five state-linked groups using models for reconnaissance, scripting and social engineering, and finding no novel AI-enabled attack. The baseline everything since is measured against.

    vendor-authored

Social engineering at scale 1

Model-generated phishing and pretexting, and the evidence about its effectiveness.

Governance and policy 4

National guidance, standards and policy on AI-enabled cyber threat.

Tooling 1

Scanners, harnesses and utilities for testing AI systems.

  • NVIDIA/garak

    NVIDIAGitHubToolOpen

    NVIDIA's LLM vulnerability scanner. Prompt injection, jailbreaks, training-data leakage, glitch tokens. Apache-2.0.

    living

Honeypots and honeynets 1

Decoy systems built to be attacked and observed.

  • LLM in the Shell: Generative Honeypots

    Sladić, M. et al.IEEE European Symposium on Security and Privacy Workshops (EuroS&PW)2024PaperOpen

    shelLM, an LLM-backed Linux shell honeypot reporting a 0.90 true negative rate against security experts asked to tell it from a real host. The paper that started the generative-honeypot line.

LLM-backed honeypots 4

Decoys whose responses are generated by a language model rather than scripted.

  • LLM Honeypot: Leveraging Large Language Models as Advanced Interactive Honeypot Systems

    Otal, H. T.; Canbaz, M. A.arXiv2024PaperOpen

    Fine-tunes an open-weights model on captured attacker sessions to generate honeypot responses, then evaluates realism and deploys it live. Preprint; evaluation is thinner than shelLM's.

    preprint

  • SoK: Honeypots & LLMs, More Than the Sum of Their Parts?

    Bridges, R. A. et al.arXiv2025Survey / SoKOpen

    Systematizes both directions at once: LLMs used to build honeypots, and honeypots built for LLM attackers. Includes a taxonomy of honeypot detection vectors and a critique of how the area evaluates itself.

    preprint

  • 0x4D31/galah

    Karimi, A.GitHubToolOpen

    LLM-powered web honeypot: generates a plausible HTTP response to whatever arrives instead of emulating fixed applications. Go, Apache-2.0.

    living

  • beelzebub-labs/beelzebub

    Beelzebub LabsGitHubToolOpen

    Low-code deception runtime, Go, GPL-3.0. The interesting part is the MCP bait tooling: decoys aimed at AI agents rather than human intruders. A commercial product sits on top.

    livingvendor-authoredopen version

Deception against machine attackers 4

Deception designed for automated adversaries rather than human ones.

Frameworks and doctrine 1

Structured references for planning and running deception programmes.

  • mitre-atlas/atlas-data

    MITRE CorporationGitHubFrameworkOpen

    ATLAS tactics, techniques, mitigations and case studies as versioned YAML. Easier to cite than the client-side-rendered website.

    livingopen version

Order effects 1

Question-order and sequence phenomena, and the QQ equality that predicts them.

Machine belief states 1

Applying human-cognition models and bias inventories to model behaviour.

  • Cognitive Bias in High-Stakes Decision-Making with LLMs

    Echterhoff, J. M.arXiv2024PaperOpen

    BiasBuster, a 16,800-prompt framework for measuring and mitigating cognitive bias in model decisions. Relevant here because deception doctrine assumes exploitable biases; this is the closest thing to an inventory of them in machines.

    preprint

Surveys and taxonomies 4

The fastest way into a literature.

Ongoing trackers 2

Sources that keep publishing on the topic.

  • Threat-Actors-use-of-Artificial-Intelligence

    cybershujinGitHubToolOpen

    Documented threat-actor AI use mapped to ATT&CK, from Microsoft, OpenAI, Anthropic, GTIG and Kaspersky reporting. Narrower than it looks: attacks on AI systems and influence operations are excluded by design.

    living

  • tl;dr sec

    Gibler, C.tl;dr secNewsletterFree registration

    Weekly security newsletter, free, with steady AI-security coverage. The feed avoids the signup.

    livingfeed

Courses 1

Structured teaching material.

  • AIRT — AI Red Team Academy

    Karimi, A.AI Red Team AcademyCourseOpen

    Eight free modules with Docker labs: prompt injection, RAG exploitation, multi-agent attacks, automated red teaming.

    living

Venues 2

Where this work is published and presented.