LLM agents sidestep CAPTCHAs; v3 trusts the browser
A new study puts bot defences against modern automation under the torch. Commodity CAPTCHA solvers beat challenge pages near perfectly and for pennies, while Large Language Model (LLM) agents pass when they bolt in a solver. Non-interactive checks like reCaptcha v3 resist, but success hinges on environment authenticity, not behaviour.
Bot management has leaned on two crutches: make automation solve a puzzle, or quietly score its behaviour. A new measurement study asks how those hold up against today’s attackers: commodity CAPTCHA solvers and Large Language Model (LLM) browser agents that can read, click and navigate like an intern with unlimited patience.
The authors built a public testbed across major defences — hCaptcha, reCaptcha v2 (checkbox and invisible), reCaptcha v3, and Cloudflare Turnstile — then ran seven commercial solver services and six LLM agents through it. Interactions were captured and judged on two things: did the bot clear verification, and did it actually submit the form.
Challenge puzzles: cheap to outsource
Challenge-based CAPTCHAs fell over. Commercial solvers achieved near-perfect bypass at roughly $0.10 to $5.00 per 1,000 solves. That is not a security control; it is a line item. LLM agents themselves mostly failed the puzzles unless they included a dedicated solver module, in which case they sailed through. If your gate depends on an image grid, your adversary depends on a cheap API key.
Trust scores: environment is the gate
Non-interactive systems told a different story. reCaptcha v3 pushed back hard on solver services, with low average success and scores often below the default 0.5 threshold. But the resistance was not about “human-like” behaviour. Two agents with nearly indistinguishable interaction traces had opposite outcomes: the one running inside a real browser profile — persistent cookies, history, stable fingerprinting — was consistently trusted. The one in a clean, instrumented environment was not. The effective security boundary sits at execution environment authenticity, not agent reasoning.
From an attacker’s perspective, the path is straightforward: host the agent where the browser looks lived-in, then add a solver module for any explicit challenges. You are no longer fighting the behaviour classifier; you are inheriting trust. That shifts effort from clever prompts to acquiring or emulating persistent browser state.
Operationally, this matches what breaks at 03:00: signup abuse and scraping do not stop at a checkbox, and “human” mouse jitter will not save you. What matters is whether the stack can tell a clean lab VM from a warm, messy profile with history and cookies attached.
There are caveats. The testbed is controlled, agents and services ran in default configurations, and thresholds followed vendor defaults. Results will vary by site and tuning. The uncomfortable constant is where the gate actually sits. If trust lives in the environment, the open question is how to bind it to something automation cannot cheaply clone.
Additional analysis of the original ArXiv paper
📋 Original Paper Title and Abstract
Broken Gates: Re-evaluating Web Bot Defenses in the Age of LLM Agents
🔍 ShortSpan Analysis of the Paper
Problem
The paper studies whether modern web bot defences remain effective against two evolving attacker classes: commercial CAPTCHA‑solving services and large language model based browser agents that autonomously navigate, reason about page content, and act on web interfaces. This matters because defenders increasingly rely on either explicit challenges or silent, trust‑based scoring to distinguish humans from automation, while attackers can outsource solving or run agentic automation that blurs the behavioural distinction.
Approach
The authors built a controlled public testbed of seven web applications on separate subdomains, each protected with one of several configurations of hCaptcha, reCaptcha v2 (checkbox and invisible), reCaptcha v3, and Cloudflare Turnstile (managed and invisible). They evaluated seven commercial solver services and six LLM‑based browser agents spanning cloud‑hosted, self‑hosted, AI‑assisted browser, and browser‑extension deployments. Agents were run in default configurations and interactions were recorded. Solvers were queried via provider APIs until 100 instances per target were collected. Agent tests used a screening phase of five attempts and, where promising, ten additional trials. Success required both passing verification and submitting the target form. The study emphasised execution environment authenticity and fine‑grained interaction trace analysis to separate behavioural signals from environment signals.
Key Findings
- Commercial solvers defeat challenge‑based CAPTCHAs with near‑perfect reliability; solving prices ranged roughly from $0.10 to $5.00 per 1,000 solves, making challenge‑based defences practically ineffective against cost‑driven adversaries.
- Solvers perform poorly against behavioural systems: average success on reCaptcha v3 was low (reported average 23%), with the best provider achieving 63% and many failing to produce verifiable tokens above the default 0.5 threshold.
- LLM‑based agents generally fail challenge‑based CAPTCHAs unless they include a dedicated solver module; cloud agents with integrated solver capabilities succeeded where self‑hosted agents without them failed.
- Non‑interactive systems fail depending on environment authenticity rather than agent behaviour. Two agents with nearly indistinguishable interaction traces produced divergent outcomes: NanoBrowser, operating inside a real browser profile with persistent cookies, history and stable fingerprinting, consistently bypassed reCaptcha v3, while Browser‑Use, running in a clean instrumented environment, received low trust scores.
- Overall, the security boundary for non‑interactive defences lies at the execution environment layer rather than purely at solver or reasoning capabilities.
Limitations
Experiments used a specific controlled testbed and default, unmodified agent configurations; results may vary with other agent customisations, solver providers, site configurations or longer operational campaigns. The selection of services and agents reflects publicly accessible systems and the authors did not develop new solvers. Behavioural thresholds follow vendor defaults such as a 0.5 reCaptcha v3 cutoff. The study intentionally omits release of exploit code and fine implementation details.
Implications
Offensive implications are clear: attackers can cheaply and reliably bypass challenge‑based CAPTCHAs using commodity solver APIs; they can also gain silent acceptance by focusing on environmental authenticity, for example by operating agents within real browser profiles or browser extensions that inherit persistent cookies, history and stable fingerprints. Combining third‑party solvers, LLM agents and methods to emulate or obtain persistent browser state lets adversaries evade both puzzle centric and behaviour‑scoring defences, shifting attacker effort toward acquiring authentic execution contexts rather than improving pure reasoning capability.
