VibeGuard stops AI-generated leaks before publish
Defenses
Developers are leaning on AI assistants to write and even package code, often accepting the result with little review. The authors call this vibe coding. It works until it does not. On 31 March 2026, an npm package for Claude Code CLI shipped a 59.8 MB source map that exposed about 512,000 lines of proprietary TypeScript. The culprit was not a logic bug but a packaging rule that let sensitive material slip into the artefact. Conventional static analysis and secret scanners did not catch it, because they were never looking at the final package contents.
Enter VibeGuard, a pre-publish security gate that inspects both the project tree and the built artefact before anything goes live. The idea is simple and refreshingly pragmatic: test the thing you are actually about to ship. The framework groups checks into five buckets that map neatly to common AI-era slip-ups. An artefact scanner looks for anomalous files such as private keys, .env files, IDE directories and oversized junk. A configuration scanner inspects packaging controls in files like package.json, .npmignore and Dockerfile, looking for drift from intended publish-time invariants. A source-map scanner parses .map files and flags cases where the sourcesContent field is populated, which indicates full source disclosure rather than benign mappings. A secret scanner uses regular expressions with placeholder filtering to reduce noise. A dependency scanner highlights unpinned versions, missing lockfiles, install hooks and insecure URLs.
Findings carry severity, the relevant Common Weakness Enumeration (CWE) where applicable, locations and remediation tips. A policy engine then renders a pass or fail decision under one of three modes: default, strict or permissive. The authors position VibeGuard to run where it hurts most if ignored: at publish time. That can be a continuous integration/continuous delivery (CI/CD) check, a prepublishOnly hook, or even a scan triggered by an assistant before it suggests a publish command.
How it performed
In controlled tests on eight synthetic projects, seven with planted issues and one clean, VibeGuard detected every seeded problem. That is 100% recall. Precision came in at 89.47%, for an F1 score of 94.44%. The policy-driven gate decisions were correct for all eight projects across default, strict and permissive modes. The nuance around source maps is worth calling out: not every .map file is bad. Only those with sourcesContent populated amount to a source disclosure. VibeGuard earns points for parsing the JSON rather than blocking .map files outright.
Why put the gate at publish time? Because pre-commit hooks and version-control checks only see source. Many of these exposures arise during build or packaging. A clean repository can still produce a leaky tarball.
Caveats and implications
The evaluation used synthetic projects, so the 100% recall may be an upper bound until tried against messy real-world repositories. The secret scanner is pattern-based, which can miss obfuscated or bespoke formats and may throw a few benign matches. The framework is static; it will not catch runtime-only issues. Coverage today focuses on npm and pip ecosystems, with others left for future work. It also does not try to distinguish AI-generated from human-authored files.
Even with those limits, the contribution is sensible: treat the publish step as a security boundary and formalise checks that reflect today’s failure modes. The emphasis on absence checks and clear remediations fits teams who have added AI into their build room. For security leads juggling speed and safety, a lightweight, policy-backed gate on the artefact itself is an easy win that could have prevented the source-map leak that motivated this work.
Additional analysis of the original ArXiv paper
📋 Original Paper Title and Abstract
VibeGuard: A Security Gate Framework for AI-Generated Code
🔍 ShortSpan Analysis of the Paper
Problem
The paper studies a class of operational security failures that arise when developers delegate code and configuration to AI assistants and skip detailed review, a practice the authors call vibe coding. It was motivated by a high-profile incident in which a shipped npm package included a 59.8 MB source map containing roughly 1,900 files and about 512,000 lines of proprietary TypeScript. The root cause was a packaging misconfiguration rather than a logic bug, revealing a gap: existing static-analysis, secret scanners and dependency auditors focus on code-level or known-CVE issues and do not inspect what actually ends up in published artifacts.
Approach
The authors present VibeGuard, a pre-publish security gate designed to inspect the project tree and the contents of the build artefact before publishing. VibeGuard comprises five scanner modules mapped to a five-category taxonomy: ArtifactScanner (detects anomalous files such as private keys, .env files, IDE directories, large unexpected artefacts), ConfigScanner (checks packaging controls like package.json, .npmignore, Dockerfile and other invariants), SourceMapScanner (parses .map files and flags populated sourcesContent as source disclosure), SecretScanner (regular-expression based detection for common credential formats with placeholder filtering) and DependencyScanner (flags unpinned versions, missing lockfiles, install hooks and insecure URLs). Findings include severity, CWE where applicable, location and remediation suggestions. A PolicyEngine aggregates findings and applies one of three built-in policies - default, strict and permissive - to make a pass/fail gate decision. VibeGuard is intended to run at publish time and can be deployed as a CI/CD check, a prepublishOnly hook, or integrated with the AI assistant so scans occur before a publish command is suggested.
Key Findings
- VibeGuard detected every seeded vulnerability in a controlled evaluation of eight synthetic projects (seven vulnerable, one clean control), achieving 100% recall.
- Overall precision was 89.47% with an F1 score of 94.44%; most scanners achieved perfect precision except the ConfigScanner, which intentionally flags missing ignore patterns and produced the majority of false positives.
- Gate decisions were correct for all eight projects across all three policy levels; the negative control produced zero findings and passed.
- The SourceMapScanner is critical because not all .map files disclose source - only those with populated sourcesContent are full disclosures; VibeGuard distinguishes these cases by parsing JSON.
- Pre-publish scanning is necessary because pre-commit hooks or version-control checks can miss artefact-level exposures that arise during the build or packaging step.
Limitations
The evaluation used synthetic projects with planted issues, so recall may be an upper bound until validated on production repositories. The SecretScanner relies on regular expressions and a placeholder heuristic, which can miss obfuscated or non-standard secrets and may produce benign false positives. VibeGuard is static only and cannot detect runtime-only vulnerabilities. Current coverage focuses on npm and pip ecosystems; other package ecosystems require additional work. The tool does not distinguish AI-generated from human-authored files.
Why It Matters
VibeGuard addresses a neglected security boundary between a successful build and the published artefact. As vibe coding spreads, packaging misconfigurations, exposed source maps and accidental inclusion of secrets or unpinned dependencies produce high-impact leaks and supply-chain risk. A publish-time gate that emphasises absence checks and clear remediations provides a practical defence-in-depth measure for teams that rely on AI code generation, and can be integrated into AI assistants and CI/CD pipelines to reduce incidents like the cited source-code leak.
