AI-generated code needs runtime testing, not static trust alone

At a glance

What this is: This is an analysis of how AI-assisted development changes application security, with a key finding that runtime testing is necessary because LLM-driven behaviour is non-deterministic.

Why it matters: It matters to IAM and security teams because AI-generated code and AI-powered APIs can expand attack surface around authentication, authorisation, secrets, and downstream access control.

By the numbers:

• 77% of respondents said AI tools improve code security, while 59% are concerned that LLM-trained tools will introduce vulnerabilities that appeared in the training code.
• 6 of the 10 OWASP Top 10 for LLM risks are expected to benefit from DAST's core testing concept, according to the article's analysis.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope.

👉 Read StackHawk's analysis of secure code testing in the age of AI

Context

AI-assisted coding changes the security problem from static code review alone to continuous verification of what applications actually do at runtime. In AI Security, the core issue is not simply that code is generated faster, but that outputs can vary, hidden dependencies can be introduced, and vulnerable patterns can arrive through copy and paste or model output. For identity and access programmes, that means authentication flows, secrets handling, and permission boundaries need to be tested as behaviours, not assumed from source review.

This matters because AI-generated software often touches identity controls even when the article is framed as application security. APIs, tokens, service accounts, and delegated access all sit inside the path from prompt to running application, so defects in authorisation or secret handling can become immediate exposure points. That starting position is typical for modern AI application teams, and it is increasingly common across cloud-native delivery.

StackHawk's angle is that DAST becomes more relevant when applications are shaped by generative AI because the thing being tested is runtime behaviour, not just source code. That is a practical reminder for programmes that already rely on SAST, because AI-generated code can look clean while still behaving unsafely once inputs, outputs, and integrations are exercised.

Key questions

Q: How should security teams govern AI-generated code in production environments?

A: Security teams should treat AI-generated code as normal production code with extra provenance risk. Require architectural review, test coverage, static analysis, and approval before merge. Then bind the agent and the build pipeline to least privilege, short-lived credentials, and complete audit logging so implementation speed does not outrun control.

Q: Why does AI-assisted development increase security risk even when developers use familiar controls?

A: Because familiar controls often assume code behaves predictably, while LLM-influenced systems can vary by prompt, context, and input shape. That creates gaps around validation, output handling, and hidden data flows. Teams may pass SAST and still ship an application that accepts unsafe model output or exposes secrets through an API path.

Q: What do security teams get wrong about AI safety testing?

A: The common mistake is treating AI safety testing as if it were just another security scan. It is not. Safety testing is about proving how a model or agent fails under pressure, while traditional security tooling is about who can access the system. Those are different governance questions and need different evidence.

Q: Should organisations treat AI-generated code as a separate governance category?

A: Yes, because AI-generated code introduces a trust problem that sits between development, application security, and identity governance. Teams need policy for what AI may generate, how outputs are validated, and how identities and secrets are handled in the resulting code. Without that, responsibility for unsafe behaviour becomes fragmented across teams.

Technical breakdown

Why non-deterministic LLM output changes application security testing

Large Language Models do not produce identical responses for identical prompts in all cases, which makes them fundamentally different from deterministic code paths. That non-determinism means a single test case rarely proves safety. Security teams need repeated, varied input testing to see how prompts, payloads, and edge cases influence outputs. In practice, that is closer to fuzzing than to one-pass static review, because the risk is in the behaviour the application exhibits under different conditions, not just in the code text itself.

Practical implication: add repeated runtime test cases for AI-powered inputs and outputs instead of treating one clean scan as sufficient.

Why DAST matters for AI-generated APIs and web apps

Dynamic Application Security Testing inspects running applications by sending inputs and observing outputs, which is useful when AI changes behaviour after deployment. AI-generated APIs can hide insecure logic, weak validation, or unsafe data handling that source-level tools may miss. DAST does not replace SAST, but it covers the operational layer where prompts, model responses, and integration logic meet. That is where authentication errors, data exposure, and logic flaws surface in AI-assisted systems.

Practical implication: place DAST into CI/CD so AI-influenced APIs are tested before release, not only after code review.

How OWASP LLM risks map to runtime control gaps

LLM-specific risk categories often involve prompt injection, data leakage, insecure output handling, and excessive autonomy in downstream actions. Those risks are not abstract model issues alone; they become control failures when the application trusts AI output too much or fails to validate what the model returns. Runtime testing helps expose where the application accepts unsafe content, follows malicious instructions, or exposes sensitive data through the wrong path. In identity terms, the weak point is often delegated trust.

Practical implication: test how AI outputs interact with authorisation, secrets, and data-access controls before the model output can trigger real action.

Threat narrative

Attacker objective: The attacker aims to exploit unsafe AI-driven application behaviour to reach data exposure, unauthorised actions, or broader compromise.

1. Entry occurs when AI-generated code or LLM-powered APIs are introduced into the application path and begin handling user or system input.
2. Escalation happens when unsafe prompts, copied code, or unchecked model outputs bypass validation and influence runtime behaviour.
3. Impact follows when the application exposes sensitive data, weakens authorisation, or executes flawed logic in production.

NHI Mgmt Group analysis

Runtime verification is becoming the control plane for AI-assisted development. Static review still matters, but it cannot prove how LLM-influenced code behaves once prompts, inputs, and integrations are live. That shifts security assurance toward DAST, repeated test inputs, and runtime observation. For practitioners, the key conclusion is that safe AI-assisted delivery depends on behaviour validation, not code appearance alone.

AI-generated code creates identity risk before it creates model risk. The most immediate failures often sit around tokens, service accounts, and API permissions, because AI-assisted applications need access to data and services to function. Once those identities are embedded in the runtime path, weak controls can turn a coding convenience into an access problem. The practitioner takeaway is that identity review must move into AI application testing, not sit outside it.

OWASP's LLM risk language is useful only when it is tied to control failure. Prompt injection, unsafe output handling, and sensitive data leakage are operational problems when applications trust model responses too broadly. LLM behaviour blind spot: the control gap is not merely that AI is unpredictable, but that teams often have no repeatable way to observe that unpredictability before production. For practitioners, the conclusion is that observability and testing depth must match model variability.

Secure code in the age of AI is really a governance problem about trust boundaries. Development speed increases when teams use generative AI, but so does the chance that insecure code, hidden dependencies, or unsafe data paths enter production faster than human review can catch them. NIST AI RMF MEASURE and NIST CSF PR.AC both reinforce the need to verify behaviour and access boundaries, not just intentions. The practical conclusion is that AI security starts where runtime trust is proven, not where code is written.

What this signals

LLM runtime trust is now a programme-level issue, not just an AppSec test case. Teams that ship AI-assisted code without behavioural validation are creating an assurance gap between what reviewers think the code does and what it actually does. That gap matters most where prompts, APIs, and identity-bound services intersect, because access and data movement happen at runtime.

AI-generated code should be governed like a mutable trust boundary. The practical signal for practitioners is that source review, secret scanning, and developer education are necessary but insufficient on their own. If an application relies on model output to drive actions, then runtime testing, approval gates, and identity controls need to sit in the same workflow.

Model behaviour is only one part of the risk. The rest is what the application lets that behaviour touch. That is why AI security and IAM now overlap in a visible way around service accounts, tokens, and delegated access. When those identities are scoped poorly, even a correct model response can still trigger an unsafe business action.

For practitioners

• Expand DAST coverage for AI-influenced applications Add runtime tests for endpoints that accept prompts, generate responses, or call model-backed APIs, and vary payloads enough to surface non-deterministic behaviour. Treat the results as production-readiness evidence, not just vulnerability confirmation.
• Review identity and secret handling in AI pipelines Inventory API keys, service accounts, and tokens used by AI-assisted workflows, then verify that they are scoped tightly and not embedded in generated code or copied examples. Use the NHI Lifecycle Management Guide for rotation and offboarding decisions where applicable.
• Map AI application risks to OWASP LLM controls Use the OWASP Agentic AI Top 10 and related LLM risk guidance to check for prompt injection, unsafe output handling, and over-trusted model responses. Align each finding to a concrete test case so the control gap is visible in engineering workflows.
• Shift security sign-off from source review to runtime evidence Require teams to show observed application behaviour under test before release, especially where AI generates code paths or logic. Pair SAST findings with runtime observations so reviewers can see whether the implementation actually enforces the intended control.

Key takeaways

• AI-generated code changes the security assurance model because behaviour matters more than appearance.
• Runtime testing becomes essential when prompts, model outputs, and downstream application logic can combine to create hidden vulnerabilities.
• Identity controls, secrets handling, and authorisation checks must be verified inside AI-assisted delivery workflows, not assumed from static review alone.

Key terms

• Dynamic Application Security Testing: Dynamic Application Security Testing evaluates a running application from the outside to identify weaknesses that only appear under real execution conditions. It is useful for validating authentication, session handling, and API behaviour, especially where configuration and integrations change how the system actually responds to attack.
• Non-deterministic output: Non-deterministic output means the same prompt or input can produce different responses across runs. In security terms, that makes one-off validation weak because the risk lies in the range of behaviours the system may exhibit, not a single predictable result.
• AI-Generated Code Risk: AI-generated code risk is the possibility that model-produced output introduces insecure credential handling, hidden secrets, or repeated patterns that weaken control design. The risk is operational, because the output can be merged quickly and at scale before humans catch the problem.
• Metadata Trust Boundary: A metadata trust boundary is the line between tool content that can be safely consumed and tool content that must be validated before use. For agentic systems, descriptions, examples, and schemas are security-relevant inputs because they can influence decisions and trigger actions with real-world impact.

What's in the full article

StackHawk's full article covers the operational detail this post intentionally leaves for the source:

• How its DAST approach tests AI-generated APIs at runtime, including input variation and output validation patterns.
• Why the article argues SAST alone is insufficient for non-deterministic LLM behaviour in production workflows.
• The specific ways the article connects OWASP LLM risks to runtime security testing across the SDLC.
• How StackHawk frames secure code testing inside CI/CD for teams adopting generative AI.

👉 StackHawk's full post covers runtime testing, AI-generated API risk, and the limits of static analysis.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management for practitioners building modern security programmes. It helps identity and security teams connect access control, lifecycle discipline, and operational governance across human and non-human estates.