A new benchmark from University of Maryland researchers argues that many impressive scores for AI vulnerability-patching agents are measuring the wrong thing. PatchBench tests whether an agent actually removes a vulnerability and preserves normal program behavior, rather than merely stopping the one proof-of-concept input used by the benchmark from crashing.
The paper, submitted to arXiv on September 3, evaluates 11 agents on C and C++ vulnerability repair. The authors report that single-PoC validation inflates solve rates by 1.83 times on average. Their top three agents pass more than 97% of the original PoCs, but fall to about half of tasks when PatchBench adds broader security and semantic checks. For security teams considering agent-generated fixes, the gap is more important than the leaderboard position: a patch that silences one crash is not necessarily a repaired vulnerability.
The benchmark changes what counts as a successful patch
PatchBench contains 213 tasks across 16 CWE classes and 32 real-world projects. Its construction tries to remove two shortcuts that can make agent results look stronger than they are.
First, the researchers move historical vulnerabilities into newer repository contexts and mutate code around the patch site. The goal is to reduce the chance that a model can reproduce a developer fix seen in training data. Their separate similarity analysis estimates that about 25% of repository-level agent patches on SEC-bench are highly similar to historical developer patches. The authors are careful not to claim this proves training-data contamination in every case, but they treat the similarity as a meaningful threat to benchmark validity.
Second, PatchBench deliberately selects vulnerabilities whose historical fixes lie outside the crash stack. That makes it harder for an agent to pass by adding a guard at the location where the sanitizer reports a failure while leaving the real defect elsewhere in the program.
The benchmark then validates patches in layers. Security validation uses additional crashing inputs produced through fuzzing. Semantic validation checks benign inputs, sanitizer regressions, program output state and working unit tests against a reference-patched repository. A patch must remove the vulnerability without quietly changing expected behavior.
A crash disappearing is not the same as a root-cause fix
The most useful result is not that one agent beats another. It is how sharply scores fall when validation tests the behavior that a security patch is supposed to preserve.
The paper reports that the top three agents exceed 97% under single-PoC validation. With multiple security PoCs, their pass rates fall to 75% to 82%. After semantic validation is added, the strongest results are roughly half of the task set. The authors also identify 67 PatchBench tasks that none of the 11 evaluated agents solve.
Their motivating example shows why. A vulnerability can originate in one component but crash later in another. An agent that patches the crash location may stop the provided input from triggering a sanitizer error while leaving malformed state or alternate exploit paths intact. In a normal software test this can look like success. In security work it can be a false sense of closure.
That distinction matters for production workflows because vulnerability remediation is not just code generation. It is a claim about the absence of a specific unsafe behavior without unacceptable regressions.
Security teams need an acceptance gate separate from the coding agent
Aipolix's main conclusion is operational: organizations using coding agents for vulnerability repair should separate patch generation from patch acceptance.
An agent may propose a fix, but the release gate should independently ask at least three questions. Does the patch eliminate more than the originally reported exploit path? Does it preserve legitimate behavior on benign inputs and existing tests? And does the change address the root cause rather than only the crash site?
That separation changes how agentic security pipelines should be designed. A workflow that lets the same agent generate a patch and declare it fixed after replaying one PoC is structurally weak, even if the model is capable. Independent fuzzing, regression checks and behavior comparisons become part of the trust boundary.
PatchBench also highlights a benchmark-governance issue. When public historical vulnerabilities are reused, high scores may mix genuine reasoning with memorized patch patterns. Transplanting vulnerabilities and mutating patch contexts is not a perfect contamination detector, but it makes the evaluation harder to game through exact or near-exact recall.
The evidence is strong enough to matter, but not yet fully reproducible
This is still a first-version arXiv preprint, not an independently replicated production study. The experiments focus on C and C++ vulnerability patching, so the findings should not automatically be generalized to every language, bug class or enterprise repository.
There is also a release inconsistency worth making explicit. The paper states that the authors release the code and benchmark at github.com/ai-sec-lab/PatchBench. When Aipolix checked that first-party URL on September 4, it returned 404 and could not be inspected. That does not invalidate the paper's reported experiments, but it currently prevents an independent practitioner from verifying the released benchmark artifact through the advertised repository.
The right interpretation is therefore narrower than saying AI agents cannot patch vulnerabilities. PatchBench shows that common evaluation methods can materially overstate successful repair when they equate one disappearing crash with a correct security fix. Its practical challenge is to make patch validation as rigorous as patch generation.
For teams already experimenting with autonomous remediation, that is actionable now: treat agent-generated security patches as proposals until independent security and semantic validation has passed.