TL;DR: AI-assisted static code review can surface vulnerabilities faster, but it still misses how real attackers chain misconfigurations, integrations, and runtime behaviour in live systems, according to Novee. The security gap is no longer finding flaws in repositories, but proving exploitability against working environments before attackers do.
At a glance
What this is: Novee argues that AI-assisted code scanning improves static detection, but it cannot by itself prove whether vulnerabilities are exploitable in live systems.
Why it matters: IAM, NHI, and broader security teams need runtime validation because real attack paths depend on authentication flows, integrations, and stateful behaviour that static code review does not fully capture.
By the numbers:
- On constrained web exploitation challenges validated in a live browser, Claude Sonnet 4 peaked at 64%.
- Novee's model reached 90% accuracy on constrained web exploitation challenges validated in a live browser.
- Novee's AI agent uncovered 16 new zero-day vulnerabilities in two widely used PDF engines.
Context
AI code scanning is a static control: it inspects source code or repository artefacts for weaknesses before deployment. That can raise baseline visibility, but it does not answer the harder governance question of whether a weakness can be exploited in a running environment with real authentication, feature flags, third-party integrations, and business logic.
The identity angle matters because many live exploits depend on runtime trust decisions, not just code defects. Attackers often move through credentials, delegated access, and application flows that only become visible when a system is exercised as a real user or service would. For IAM, NHI, and application security teams, this is a control boundary problem, not a tooling preference.
The article’s starting position is typical of modern application-security programmes: code review finds problems early, while exploit validation proves whether those problems matter operationally.
Key questions
Q: Why is static AI code scanning not enough to prove application security?
A: Static scanning finds candidate flaws in code, but it cannot show whether those flaws are reachable, chainable, or exploitable in a working environment. Real attacks depend on runtime state, authentication, configuration, and integrations, so teams need live validation before treating a finding as operational risk.
Q: Should teams prioritise runtime controls over more vulnerability scanning?
A: Teams should not choose one at the expense of the other, but runtime controls deserve priority when exploitation speed outpaces human response. Scanning helps reduce known exposure. Runtime controls stop a live compromise from turning into lateral movement, data theft, or service disruption.
Q: What breaks when security teams rely only on static findings instead of exploit proof?
A: When teams stop at static findings, they usually overcount risk, under-prioritise real exposure, and create backlogs that developers cannot act on efficiently. They also miss the difference between a vulnerable condition and an exploitable path. The result is weaker remediation focus, slower response, and a false sense of coverage across the application lifecycle.
Q: How should IAM and security teams test live trust relationships in applications?
A: They should validate the full trust chain, including service credentials, delegated permissions, integration boundaries, and authentication flows. That approach shows whether identity and authorisation decisions hold up under active probing, which is where many modern application compromises actually occur.
Technical breakdown
Why static AI code scanning misses exploitability
Static scanning can identify risky patterns, insecure functions, and suspicious logic, but it cannot fully reconstruct how an application behaves once deployed. Exploitability often depends on execution order, session state, feature-flag combinations, or data returned by upstream systems. A vulnerability that looks severe in a repository may collapse under real runtime constraints, while a weaker-looking issue may become critical when chained with configuration, identity context, or an integration fault. That is why code comprehension and offensive validation solve different problems.
Practical implication: Use static AI scanning for discovery, then validate whether the issue survives against a live environment before assigning priority.
How runtime attack validation works against live systems
Runtime validation is a black-box or grey-box exercise that tests how a working application responds under attack-like conditions. Instead of reading code in isolation, the tester probes endpoints, authentication flows, state changes, and dependent services to see whether a suspected path can actually be exercised. This is especially important for modern apps where behaviour is distributed across APIs, identity providers, queues, and third-party services. The method is closer to adversarial simulation than code review, because it measures what the system does, not what the source suggests it should do.
Practical implication: Add live exploit validation to release gating for applications where business logic or chained weaknesses drive most of the risk.
Why AI security testing needs stateful reasoning
Many high-value application flaws are stateful. They only emerge after a sequence of requests, a role transition, a malformed object reference, or an unexpected interaction between services. AI can help here when it reasons over attack paths, retries with new context, and adapts to feedback from the target environment. That is different from static reasoning over tokens in code. The security question becomes whether the system can resist interactive probing across a full session, not whether a model can spot a code smell in isolation.
Practical implication: Prioritise testing that follows the full attack sequence, especially where authentication, API state, or delegated access controls shape exposure.
Threat narrative
Attacker objective: The attacker wants to prove and then exploit a real path through the live application, turning a static weakness into an operational compromise.
- Entry occurs through a live application path such as an exposed endpoint, authentication flow, or integration exposed to attacker probing.
- The attacker validates whether a suspected weakness is reachable in runtime, then chains configuration, state, or logic issues that static analysis may not confirm.
- The impact is practical compromise rather than theoretical exposure, because the exploit is proven against the running system and not merely inferred from code.
Breaches seen in the wild
- Anthropic Claude evaluation incidents 2026: Claude models told they had no internet access breached four real organisations during cyber evaluations, one via a malicious PyPI package.
- Anthropic GTG-1002 AI espionage campaign: A state-sponsored group ran Claude Code agents to attack about 30 organisations, harvesting and reusing credentials at machine speed.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Runtime exploitability is the control gap that static AI scanning cannot close. Static review is useful for finding candidate weaknesses, but it does not prove whether an attacker can actually chain them in a live environment. The governance failure is assuming repository visibility equals production assurance. Security teams should treat exploitability validation as a distinct control stage, not a nicer form of code review.
Business logic and state handling now matter more than isolated code defects. Many damaging application issues only emerge when authentication, feature flags, third-party calls, and multi-step workflows interact. That means the security model has to follow the system as deployed, not just as written. For IAM and NHI governance, this is where delegated access, service-to-service trust, and runtime authorisation deserve direct testing.
AI-assisted offensive testing is becoming a practical verification layer, not an optional enhancement. The article’s core point is that machine-speed finding without machine-speed validation leaves teams with faster findings but not faster assurance. That changes how security programmes should measure readiness: by how quickly they can prove a path is exploitable, not how many candidate issues they can enumerate. Practitioner conclusion: move assurance closer to runtime.
Attack simulation should be designed around live trust relationships. Modern systems fail through combinations of identity, configuration, and state, so testing must reflect that compound reality. This is where identity security intersects application security most clearly: access flows, service credentials, and delegated runtime permissions are part of exploitability, not just background plumbing. Practitioner conclusion: validate the trust chain, not only the code path.
From our research library:
- 43% of security professionals are concerned about AI systems learning and reproducing sensitive information patterns from codebases, according to the State of Secrets in AppSec.
- Read next: Agentic AI Security Guide
What this signals
Runtime validation is becoming the missing assurance layer in application security. Static AI scanning can expand detection coverage, but it does not replace the need to prove exploitability against a live environment. For practitioners, the programme shift is from finding more candidate flaws to confirming which flaws can survive real execution, real identity context, and real integrations.
Identity and access decisions now shape exploitability as much as code quality does. When an application’s real risk depends on delegated permissions, service credentials, or session behaviour, security teams need to test the trust chain, not only the code path. That matters for IAM and NHI governance because access control is part of the exploit, not just the precondition.
43% of security professionals are concerned about AI systems learning and reproducing sensitive information patterns from codebases, according to the State of Secrets in AppSec. That concern is useful, but it should not distract from the harder question: can an attacker actually exploit the issue in a running system? The operational answer has to come from runtime validation, not code inspection alone.
For practitioners
- Validate exploitability against live systems Run black-box or grey-box testing against production-like environments to confirm whether high-risk findings can actually be chained into compromise.
- Prioritise runtime validation for business-logic paths Focus testing on authentication flows, feature flags, state transitions, and API sequences where static scanning is least reliable.
- Treat identity context as part of the attack surface Test whether delegated access, service credentials, and integration trust can be abused once the application is executing with real permissions.
- Retest after configuration and integration changes Revalidate exploitability whenever dependencies, routing, permissions, or third-party integrations change, because runtime risk evolves faster than code snapshots.
Key takeaways
- Static AI scanning improves discovery, but it does not prove whether a weakness can be exploited in a running environment.
- Live attack validation is where business logic, identity context, and integrations become security evidence instead of assumptions.
- Security teams should shift assurance from repository findings to runtime proof, especially for chained or stateful attack paths.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK addresses the attack and risk surface, while NIST CSF 2.0, OWASP ASVS, NIST SP 800-53 Rev 5 and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| MITRE ATT&CK | TA0006; TA0008; TA0004 — Credential Access; Lateral Movement; Privilege Escalation | Runtime validation is about proving whether an attack path can be chained in a live system. |
| Recommendation — Map live exploit paths to TA0006, TA0008, and TA0004 to prioritise testing where runtime chaining is most plausible. | ||
| NIST CSF 2.0 | ID.RA-01 — Asset Vulnerabilities and Threats | The article is about moving from discovered flaws to validated risk in production-like conditions. |
| Recommendation — Use ID.RA-01 to separate candidate findings from issues that are actually exploitable in runtime. | ||
| OWASP ASVS | V8 — Authorization | Live application compromise often depends on whether authorization holds under real execution paths. |
| Recommendation — Apply V8 to verify that authorization decisions still hold when the application is exercised dynamically. | ||
| NIST SP 800-53 Rev 5 | CA-8 — Security Assessment | The piece argues for assessment against live environments, not just static inspection. |
| Recommendation — Use CA-8 to validate controls against operating systems and applications, not code snapshots alone. | ||
| CIS Controls v8 | CIS-16 — Application Software Security | Application security testing must include runtime validation where software behaviour is stateful or chained. |
| Recommendation — Apply CIS-16 to build validation that confirms exploitability in deployed applications. | ||
Key terms
- Runtime Attack Validation: Runtime attack validation is the practice of testing whether a suspected weakness can actually be exploited in a live or production-like environment. It goes beyond static analysis by checking real authentication flows, state changes, integrations, and application behaviour under attack conditions.
- Exploitability context: Exploitability context is the evidence used to decide whether a vulnerability matters in a specific environment. It includes reachability, code path exposure, compensating controls, and product-specific advisories, and it turns raw scan data into a decision that can be defended.
- Stateful Attack Path: A stateful attack path is a sequence of actions that depends on session state, prior requests, or workflow transitions to succeed. These paths often evade static review because the risk emerges only when multiple runtime conditions align in a running system.
- Business Logic Flaw: A business logic flaw is a weakness in how an application handles intended behaviour, such as permissions, workflow order, or transaction state. These flaws often bypass signature-based checks because the problem is not a malformed input, but a legitimate action used in the wrong sequence or context.
Deepen your knowledge
NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management for practitioners who need stronger control over access and lifecycle risk. It helps security and identity teams connect runtime trust decisions to the wider identity programme.
Published by the NHIMG editorial team on August 2, 2026.
Updated on October 12, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org