URL-only black-box DAST reached 51 of 60 apps in Argus testing

At a glance

What this is: NHI Mgmt Group's analysis, drawing on ProjectDiscovery's blog post: URL-only black-box DAST on the Argus benchmark solved 51 of 60 vulnerable web applications, showing that agent evaluation depends on isolation, validation, and budget discipline.

Why it matters: This matters because security teams will use agentic testing to find web vulnerabilities, and poor benchmark hygiene can make false confidence look like capability.

By the numbers:

• Neo solved 51 of 60 vulnerable web applications in Argus, which is 85% of the benchmark.
• The average cost per solve was about $3.40.
• The benchmark cap was $20 per challenge.

Context

Black-box DAST depends on what the agent can infer from runtime behaviour, not on source code or architecture diagrams. That makes the testing model useful for exposed web applications, but it also means benchmark design, environment isolation, and validation logic shape the result as much as the agent itself.

For AI-assisted security testing, the real governance problem is not whether an agent can find some flaws. It is whether the evaluation environment prevents contamination, false positives, and cross-run leakage so the score reflects actual capability rather than lab artefacts. application security remains the relevant discipline because the target is still web-application attack surface.

Key questions

Q: How do black-box DAST agents work against live web apps?

A: Black-box DAST agents interact with a running application using only observable responses, then iteratively test inputs, states, and workflows until they find an exploitable path. They do not need source code, but their effectiveness depends on how much the target reveals at runtime and how well the agent can reason from that feedback.

Q: Why do vulnerable web app benchmarks need isolation and reset controls?

A: They need isolation and reset controls so each benchmark starts from a clean application, database and session state, and one test does not poison the next. Without container isolation, snapshot rollback or database reseeding, cached credentials, altered records or changed feature flags can distort results and create false positives or missed vulnerabilities.

Q: What is the difference between SAST and DAST for security teams?

A: SAST inspects code and binaries before execution, while DAST attacks the live application from the outside. SAST is better for early defect discovery, and DAST is better for runtime exposure testing. Most mature teams need both because they answer different questions about risk.

Q: What should teams do first after a DAST run produces a likely exploit?

A: The first step is to preserve the evidence and contain the runtime state, then validate the finding in a controlled copy of the target before any remediation. Teams should capture the request, response, affected endpoint and session context, while preventing the exploit path from spreading or mutating data during verification.

Technical breakdown

How black-box DAST agents infer exploitable paths

Black-box DAST operates with only a live endpoint, request/response feedback, and whatever behaviour the application reveals at runtime. Without source code, the agent has to infer authentication flows, hidden parameters, state changes, and abuse paths from observable effects. That makes it closer to iterative probing than static analysis. The benchmark therefore measures whether an agent can build a working exploit path from interaction alone, not whether it can inspect implementation details. See the OWASP ZAP Guide.

Practical implication: Treat URL-only DAST as a runtime discovery mechanism and validate it against live application behaviour, not code review assumptions.

Why benchmark hygiene changes the result

A DAST benchmark is only as reliable as its isolation model. Pinned dependencies prevent patched packages from silently removing intended vulnerabilities, per-run reset stops stored state from contaminating later attempts, and network segmentation keeps the agent from wandering into unintended services. Dynamic flag generation and exact-value verification also matter because format checks can be gamed by guessed outputs. Without those controls, the benchmark measures environmental drift instead of agent capability. See the Application Security Guide.

Practical implication: Build per-run isolation, reset state, and exact outcome validation into every evaluation pipeline.

Why budgeted evaluation is different from timed evaluation

A fixed cost budget changes what success means. Instead of rewarding brute-force persistence within a time window, it forces the evaluator to account for tool calls, retries, and inefficient exploration. That is especially relevant for agentic security workflows, where poor search strategy can consume budget without improving coverage. Cost-based stopping also surfaces a practical question: which vulnerabilities are reachable with economical exploration, and which require deeper reasoning or better tooling?

Practical implication: Use spend limits alongside success rates so you can separate efficient exploitation from expensive trial-and-error.

Threat narrative

Attacker objective: The objective is to convert limited external web access into a validated exploit path that reaches the challenge flag.

1. Entry begins with a live web application and a minimal prompt, so the agent starts from only the exposed URL and observable responses.
2. Credential or access abuse is not the main theme here, but the benchmark includes authentication and authorisation bypass cases that reward state manipulation and request replay.
3. Escalation comes from chaining behaviours such as SSRF, multi-step workflows, and application logic abuse into a flag-producing path.
4. Impact is the successful extraction of the challenge flag, which proves the agent can turn runtime access into a working exploit path.

Breaches seen in the wild

• LiteLLM PyPI package breach: LiteLLM PyPI supply chain attack, credentials stolen from users.
• LiteLLM MCP auth bypass 2026: An exploited LiteLLM MCP auth bypass and default sk-1234 master keys let attackers steal AI gateway master and provider API keys.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

URL-only DAST is becoming a governance problem, not just a testing trick. Once an agent can probe live applications without source code, security leaders have to decide whether the output is trustworthy enough to influence remediation priority. That shifts DAST from a simple tool selection issue to a question of evaluation integrity, repeatability, and false-positive control. The control failure is not detection alone, but confidence in what detection actually means.

Benchmark contamination creates a hidden measurement failure. If one run can alter passwords, poison state, or leave payloads behind, later runs no longer measure the same environment. That is especially important for agentic testing because stateful workflows make the benchmark itself part of the attack surface. Practitioners should treat environment reset and isolation as first-class governance requirements, not lab housekeeping.

Detection quality now depends on exact verification, not just exploit generation. A model that can produce plausible output is not the same as a system that proves exploitation against the build-time truth. This is the same distinction security teams face when they trust pattern matching over deterministic validation. The named concept here is validation drift: when a test harness slowly stops proving what it claims to prove. Teams need to police the harness as tightly as the target.

Agentic testing will favour disciplined search over raw model size. ProjectDiscovery's results show that a cheaper model can handle a large share of the workload while stronger models pick up the harder chains. That tells practitioners that orchestration, routing, and stopping criteria matter as much as model capability. The operational question is how to spend budget where reasoning depth is actually needed, rather than burning it on weak exploration.

Black-box coverage will remain uneven across vulnerability classes. Modern stacks, multi-service flows, and chained flaws are where agent testing earns its keep, while simple scanning still has diminishing returns. That means security programmes should use DAST agents to widen coverage, but not to replace human review of complex logic, exploitability assumptions, or post-finding triage. The right posture is coverage augmentation, not automation theater.

From our research library:

• Internal repositories are 6x more likely to contain secrets than public ones (32.2% vs 5.6%), contradicting the assumption that private repos are safe, according to the State of Secrets Sprawl 2026.
• Read next: OWASP ZAP Guide

What this signals

Validation drift: benchmark harnesses can quietly stop measuring what they claim to measure when state, dependencies, or verification rules are allowed to drift. That is especially dangerous in agentic testing, where one contaminated run can misstate capability across an entire model pipeline.

Security teams should expect more agent-driven DAST to enter normal workflow discussions, but the operational question will shift toward harness governance, not model hype. The real decision is whether the programme can prove repeatability, isolate state, and separate genuine exploit discovery from expensive search noise.

Internal repositories are 6x more likely to contain secrets than public ones (32.2% vs 5.6%), according to the State of Secrets Sprawl 2026. For practitioners, that means runtime testing should sit alongside source-controlled secrets hygiene, not as a substitute for it.

For practitioners

• Implement per-run environment isolation Reset application state, workspace memory, and network reachability for every evaluation so one run cannot influence the next.
• Use exact-value validation for findings Validate flags, proof-of-exploit markers, or other outcomes against the build-time ground truth instead of format checks or pattern matches.
• Pin benchmark dependencies before testing Lock challenge dependencies to known vulnerable versions so patch drift does not erase the flaw you are trying to measure.
• Separate cost budgets from time limits Track spend per solve and stop on budget exhaustion so inefficient agent behaviour is visible even when execution remains technically possible.
• Review failed trajectories for tooling gaps Use near-miss runs to identify whether the agent lacks a browser action, a replay step, or a better way to reason across multi-service applications.

Key takeaways

• URL-only DAST can still reach meaningful web-application coverage, but the result is only trustworthy when the test harness is isolated and exact.
• The benchmark evidence points to a measurement problem as much as a capability problem, because state contamination and dependency drift can distort outcomes.
• Security teams should treat agentic testing as a governed workflow with validation, reset, and budget controls, not as an unconstrained discovery exercise.

Related NHI Mgmt Group resources: Application Security Guide · OWASP ZAP Guide · AI Coding Agents Security Guide · Red Teaming AI Agents for Identity Abuse

Key terms

• Black-Box DAST: Black-Box DAST is dynamic application security testing performed without source code, internal design knowledge or trusted credentials. It probes a running web application from the outside, typically by sending HTTP requests and observing responses, redirects, errors and side effects to find exploitable flaws in live behaviour.
• State Contamination: State contamination is when one security test, scan or exploit attempt changes an application’s runtime or data state and affects later checks. It commonly involves modified sessions, mutated records, caches or feature flags, which can make later results unreliable or conceal the original vulnerability.
• Dynamic Flag Injection: Dynamic flag injection is the insertion of runtime flags or parameters into an application, test or exploit path to change behaviour without rebuilding code. Practitioners use it to toggle features, steer requests or alter test conditions in a live environment, often through headers, environment variables or request parameters.
• Multi-Step Exploitation Chain: A multi-step exploitation chain is an attack that combines several separate actions or weaknesses into one working compromise. An attacker may first gain limited access, then escalate privileges, move laterally or extract data, with each step depending on the previous one to reach the final impact.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity controls to the wider security programmes they run every day.