Random ramblings of a security nerd

The AIpocalypse or how LLM-based exploitation is the new normal

In the last 3-4 months, AI models have made an immense jump in exploitation capabilities. Several talks and blog posts highlight the "new" capabilities of frontier AI models. The agents have learned from countless CTF writeups, research papers on exploitation techniques, and conference talks/demonstrations on how to automate diverse techniques.

In an agentic workflow, these models are incredibly skilled at advanced exploitation. The key finding is that they render tedious mitigations useless as the agent can incrementally improve the exploit on its own thereby weaponizing proof-of-concept crashes into full chains.

"Early" LLM systems (i.e., in 2025) were used as a static analysis. Essentially telling the chat bot to find bugs in a given file and to write a vulnerability report. This worked, sometimes, very well but, most of the time, produced AI slop that was not instantiateable. Bug bounty programs were flooded with this unvalidated slop and many open source developers spoke out against this type of contribution.

But with the rise of agentic workflows, LLMs can leverage feedback to improve. A simple workflow, similar to the one presented by Nicholas in his unprompted talk breaks down the exploit synthesis into several steps.

Seeding reports: The first step is to seeding potential bug candidates and asking the first agent to go, one module at a time, one file at a time, through the source base and to find potential security bugs. One may have to tell it that this is for defense purposes to side step some ethic mitigations but generally the LLMs comply well with this task. This will result in hundreds of mostly incomplete vulnerability reports for the average project. Instead of submitting them to bug bounty programs and DoSing developers, the next agent improves the reports. This step is most like a static analysis.

Reaching locations: The second agent instantiates the PoCs. Based on the vulnerability reports, the agent tries to infer a path that reaches the bug location. This will result in a few vulnerability reports with reachable bug locations and a set of false positives. This under-approximation already leverages a concrete execution environment in which the agent can validate its findings. This step is comparable to a poor man's symbolic execution engine.

Triggering the bug: The third agent tries to trigger the bug based on probable bugs from the previous steps. The advantage is that the earlier agent has already validated that the location is reachable and this agent can not concentrate on the mutation of the seed to trigger the bug. This step results in a few validates crashes and is akin to a fuzzer.

Exploitation: The fourth step takes the initial PoC crashes, analyzes them and bypasses any deployed mitigations. Some of the frontier models try to hold back in this step and one may have to convince its agent that they are playing a CTF.

The key advantage of LLMs and AI in this environment is the automation. This simple four-stage pipeline combines code review (static analysis), path synthesis (symbolic execution), seed mutation (fuzzing), and exploitation (bypassing mitigations). All of these steps required significant manual analysis before and are now promptable.

In their current state, LLM agents favor attackers. On one hand, this will destroy the market for exploits as they now becomes a cheap commodity. On the other hand, developers will be flooded with validated bug reports. Initially, this will be tough but hopefully, as projects embrace LLM-based bug search, we will see an improvement in code quality. The area I'm most excited about is how to further improve defensive capabilities and, most essentially, how to automate patching of the discovered bugs. Let me know if you have ideas!

Other articles

The Fuzzing Hype-Train: How Random Testing Triggers Thousands of Crashes

Published: Mon 01 April 2019
By Mathias Payer

In Security.

tags: Fuzzing

Software contains bugs and some bugs are exploitable. Mitigations protect our systems in the presence of these vulnerabilities, often stopping the program when detecting a security violation. The alternative is to discover bugs during development and fixing them in the code. Despite massive efforts, finding and reproducing bugs is incredibly …
read more
SMoTherSpectre: transient execution attacks through port contention

Published: Wed 06 March 2019
By Atri Bhattacharyya Babak Falsafi Mathias Payer

In Security.

tags: SMoTher port contention speculation side channel

Side channel attacks such as Spectre or Meltdown allow data leakage from an unwilling process. Until now, transient execution side channel attacks primarily leveraged cache-based side channels to leak information. The very purpose of a cache, that of providing faster access to a subset of data, enables information leakage. While …
read more
A journey on evaluating Control-Flow Integrity (CFI): LLVM-CFI versus RAP

Published: Wed 26 December 2018
By Mathias Payer

In Security.

tags: CFI LLVM-CFI RAP

This post started out of the need to provide a little more clarification after a long and heated discussions on Twitter (initial discussion and follow up) about the origins of Control-Flow Integrity (CFI), the contributions of academia, and the precision, performance, and compatibility of different existing implementations.

CFI is a …
read more
AMD SEV attack surface: a tale of too much trust

Published: Thu 22 September 2016
By Mathias Payer

In Security.

tags: privacy encrypted memory runtime monitor

AMD recently announced the new Secure Encrypted Virtualization (SEV) extension that intends to protect virtual machines against compromised hypervisors/Virtual Machine Monitors (VMMs). An intended use-case of SEV is to protect a VM against a malicious cloud provider. All memory contents are encrypted and the cloud provider cannot recover any …
read more
Control-Flow Integrity: An Introduction

Published: Tue 13 September 2016
By Mathias Payer

In Security.

tags: defense runtime monitor CFI

At a high level, Control-Flow Integrity (CFI) restricts the control-flow of an application to valid execution traces. CFI enforces this property by monitoring the program at runtime and comparing its state to a set of precomputed valid states. If an invalid state is detected, an alert is raised, usually terminating …
read more
Reversing JS email malware

Published: Sun 08 February 2015
By Mathias Payer

In Security.

tags: malware reversing security

Another lazy Sunday (oh well, actually I should be writing papers and grant proposals but we are not talking about that right now) and I'm scrolling through my email when I stumbled upon a "FedEx notice" with your usual "you have not picked up your package" scam and I figured …
read more
On differences between the CFI, CPS, and CPI properties

Published: Tue 07 October 2014
By Mathias Payer, Volodymyr Kuznetsov

In Security.

tags: CPI CFI code reuse

At OSDI'14 we published our paper on [1] where we introduce two new security properties that protect programs against control-flow hijack attacks enabled by memory corruption vulnerabilities. The design space in this area is already very cluttered and we use this blog post to highlight the differences between the individual …
read more
A walkthrough for a difficult point and click adventure or deleting a GApps domain and all Google services

Published: Wed 22 January 2014
By Mathias Payer

In Security.

tags: cloud privacy security self-hosted

Deleting an old GApps account can result in infinite pain. Here's why.
read more
Having phun with Symbolic Execution (SE)

Published: Tue 14 January 2014
By Mathias Payer

In Security.

tags: symbolic execution security howto introduction

An introduction article that explains what symbolic execution is and how it can be chained to trigger vulnerabilities hidden deep inside binaries.
read more

Other articles

links

social