LLM triage that must cite the code finds real bugs
Reviewing VulnSAGE: Adjudicating SAST Reports by Connecting LLM Reasoning to Program Structure · Jinbao Chen, Zixiang Zhou, Yu Zhang et al.
Researchers make an LLM justify every static analysis verdict with program relations that a checker verifies, then generate inputs that reproduce the bugs. It turns tens of thousands of noisy alerts into a short list of confirmed defects, which is useful to maintainers and equally useful to anyone hunting for exploitable code.
Five static analysis tools raised 45,440 alerts across the most-starred C and C++ projects on GitHub. Nobody reads that many. The researchers behind VulnSAGE pointed an LLM pipeline at the pile and came out with 104 defects that reproduce at runtime. Maintainers who replied confirmed nearly all of them, including several that were previously unknown.
Static analysis flags operations whose safety depends on context: a buffer copy that is fine if the length was checked three functions earlier. The tool cannot see that far, so it warns and a human decides. An LLM can read the context. But ask one whether a warning is real and you get a fluent opinion with nothing to check it against.
The clever part is that the model is not allowed to just give a verdict. Every judgement must cite the program relations it rests on: this value flows from here, this helper acts on that caller object, this check guards that call. Those citations point into a graph of the program built by conventional analysis. A checker then queries the graph to confirm each cited relation exists. Facts with false premises are discarded, and the verdict comes only from what survives.
That matters more than it sounds. When the researchers bypassed the checker, accuracy dropped on both datasets they tested. The model had been citing structure that was not there. Handing it the right slice of code up front helped too, because the decisive line often sits nowhere near the reported one.
The checker verifies premises, not judgement
Whether a bounds check is sufficient is still the model's call. The checker only confirms the check exists where the model says it does. The researchers are candid about this. Analysis also runs out where it always does: macros, indirect calls and behaviour inside external libraries can leave relations unresolved. Reproduction needs a revision that builds and an environment to run it in. And public code may have been in the model's training data, though the structural check does not depend on the model.
Still, the evidence that counts is reproduction. Under the same proof-of-concept procedure, roughly eight times as many of its positive verdicts reproduced as for an earlier LLM-only adjudicator. A crash is harder to argue with than a confidence score.
What I find most interesting is what this pipeline actually is. It takes a mountain of noisy warnings, keeps the real ones and writes inputs that trigger them. For a maintainer that is triage. For an attacker it is target selection with a working proof of concept attached, and the researchers acknowledge it could be adapted that way.
If your static analysis output is ignored because it is mostly noise, I would stop counting on that. The noise was a kind of protection. It is getting thinner.
Further analysis
Problem
Static Application Security Testing reports flag code locations whose safety depends on project-specific context. Maintainers must decide which reports indicate real defects, but the decisive code often lies far from the reported line and the model's verdict requires a basis that program analysis can check. Existing LLM triage can interpret context but typically lacks a verifiable connection between the model's reasoning and concrete program structure, limiting reproducibility and auditability.
Approach
The paper presents VulnSAGE, a system that connects LLM reasoning to program structure in both directions. It builds a Program Structure Evidence Graph (PSEG) that encodes dependencies and explanatory relations (data flow, control, call bindings, declarations and type/member links) with preserved source identity. A report-centered slicer traverses PSEG from reported locations to recover the context the model needs. The model emits CLAIMs, an intermediate representation in which each semantic fact pairs an interpretation (TP or FP) with the concrete PSEG relations it assumes as premises. A structural admission checker validates all cited relations over PSEG and only admitted facts enter an evidence ledger that computes the static verdict. Statically supported true positives (TP-S) are handed to a proof-of-concept (PoC) generator and runtime runner; correlated sanitizer reports or crashes promote TP-S to dynamically reproduced TP-D. The implementation uses Joern-based C/C++ code-property graphs extended into PSEG, LangGraph orchestration and GPT-5.5 for the LLM components.
Key Findings
- On 187 labelled reports from four corpora across five vulnerability families, VulnSAGE rejects 89 of 97 false alarms and retains 85 of 90 real defects, and decides at least 96% of reports in each set.
- Report-centered pre-slicing improves adjudication accuracy by 14.81 to 27.08 percentage points; ablation without pre-slicing reduced accuracy for D2A-libtiff from 87.50% to 60.42% and for SATE-V from 92.59% to 77.78%.
- Structural admission prevents unsupported facts from deciding: bypassing admission raised false positives and false negatives and lowered accuracy to 70.83% and 88.89% on the two datasets.
- Under a common PoC procedure, 25 of VulnSAGE's positive verdicts reproduce at runtime versus 3 for an LLM-only adjudicator; applied to 45,440 alerts from five SAST tools on 200 popular C/C++ projects, it reproduces 104 defects and reduces alerts to 486 candidate defects after merging shared root causes.
- Maintainers confirmed 24 of 26 reproduced defects they responded to, including ten previously unknown defects, nine of which have been fixed.
Limitations
Coverage depends on available upstream reports and PSEG relations for the matching revision; macro expansion, indirect calls or external-library behaviour can leave relations unresolved. The model provides semantic judgements that program analysis cannot in general decide, so correctness depends on both model interpretation and admitted structural premises. Reproduction requires a buildable revision and an execution environment; broader environment and input-format support would improve recall. Prior exposure of public code during model training may affect semantic judgements, though admission checks cited relations independently of the model.
Implications
Because VulnSAGE produces admitted, source-referenced facts and automatically generates PoCs that reproduce defects, the same technique could be used offensively to prioritise and validate real vulnerabilities at scale and to create executable inputs that demonstrate or assist exploitation. Its ability to reduce large alert volumes to a compact set of verified, reproducible defects increases the efficiency of locating exploitable code paths and producing working PoCs, which raises the risk that such pipelines could be adapted by malicious actors if misused.
The paper title, authors and abstract
VulnSAGE: Adjudicating SAST Reports by Connecting LLM Reasoning to Program Structure
Static Application Security Testing (SAST) reports flag operations whose safety depends on project-specific context, and maintainers must decide which reports describe real defects. Large language models (LLMs) can interpret this context, but adjudicating a report with them requires the decisive code, which often lies far from the reported line, and a decision basis that program analysis can check. We present VulnSAGE, an adjudication system for SAST reports that connects LLM reasoning to program structure in both directions. Its report-centered slicer recovers the dependencies of the reported operation together with the program structure that explains them, such as the caller object on which a helper acts. CLAIM, an intermediate representation for report judgments, records each model interpretation as a semantic fact that cites the program relations it assumes. Program analysis checks these premises, and the verdict follows from the facts whose premises hold. Both mechanisms share a program graph, which lets the model cite the code it reads and lets the checker query those citations. Statically supported true positives then guide proof-of-concept generation and runtime reproduction. Across 187 labeled reports from four corpora covering five vulnerability families, VulnSAGE rejects 89 of 97 false alarms and retains 85 of 90 real defects. Under a common PoC procedure, 25 of its positive verdicts reproduce at runtime, compared with three for the LLM-based adjudicator LLM4SA. Applied to 45,440 alerts raised by five SAST tools on the 200 most-starred C/C++ projects on GitHub, it reproduces 104 defects at runtime. Maintainers have confirmed 24 of the 26 defects they have responded to, including ten previously unknown defects, nine of which are already fixed.
