AI-generated code often works while remaining insecure in production

At a glance

What this is: OXSecurity says AI-generated code often works in testing while still failing basic security expectations at runtime, including access control, secret handling, and dependency hygiene.

Why it matters: IAM and security teams need to treat AI-assisted development as a validation problem, because functional correctness does not prove authorisation, authentication, or NHI-relevant runtime controls are in place.

By the numbers:

• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap, according to the State of Secrets in AppSec.

Context

AI-generated code changes the security problem from whether code can be produced to whether its runtime behaviour can be trusted. The core gap is that AI systems optimise for functional patterns, while security teams need evidence for access control, input validation, secrets handling, and dependency provenance.

That creates an identity and access governance issue as much as an application security issue. When generated code touches database permissions, authentication flows, API tokens, or service credentials, the question is whether the application enforces the right access boundaries after deployment, not whether it merely runs successfully.

Key questions

Q: How should teams verify AI-generated integration, build, and infrastructure code before it reaches production?

A: Teams should apply the same independent analysis to AI-generated code that they use for hand-written code, especially in integration flows, build scripts, and infrastructure automation. The goal is to catch bugs, vulnerabilities, and maintainability issues before merge. Deterministic verification matters because a language model cannot reliably validate its own output, and silent defects in these layers can cascade quickly into production incidents.

Q: Why do AI-generated applications increase the risk of security misconfiguration?

A: Because models optimise for working output, they often emit permissive settings that satisfy the prompt but weaken the environment. That includes broad network exposure, weak storage permissions, and unsafe defaults in infrastructure-as-code, which can be exploited even when the rest of the application looks correct.

Q: What are the signs that AI-generated code is degrading security instead of improving it?

A: Common warning signs include missing input validation, weakened access controls, stripped logging, insecure API parameters, and refactors that change control flow without a security rationale. Teams should also watch for code that looks cleaner but loses redundancy, error handling, or architecture-specific safeguards. If a fix appears to work yet was not reviewed in context, treat it as a likely regression candidate.

Q: When should engineering teams keep human review in the approval path for AI-assisted code changes?

A: Teams should keep human review whenever the model cannot consistently apply blocking criteria, especially for security-sensitive changes, policy-driven reviews, or cases with ambiguous risk. Human oversight is still needed when an approval decision depends on organizational context, because a model may surface issues without correctly deciding whether they should block merge to main.

Technical breakdown

Why functional code still fails security validation

AI coding models are trained to produce plausible and often effective patterns, not to reason about threat models. That means they can generate working login flows, database queries, and API handlers that still rely on string concatenation, incomplete validation, weak crypto, or missing authorisation checks. Functional testing confirms expected behaviour with normal input, but security defects usually appear only when input is malicious or when runtime configuration is wrong. The result is a false sense of readiness: the feature behaves correctly, yet the control plane that should constrain access never existed or never activated.

Practical implication: test adversarial inputs and runtime policy enforcement, not just happy-path functionality.

Why static tools miss AI-generated runtime risk

Static analysis can inspect code, but it cannot always prove whether the surrounding controls are actually active. That matters when a generated application depends on database policies, cloud storage permissions, token validation, or service-to-service authorisation that lives outside the source file. AI-generated applications can pass scans while remaining exposed because the control failure sits in the environment, not the code line. This is especially true when generated infrastructure and application logic are produced together, since the security state emerges from how those parts interact after deployment.

Practical implication: pair code scanning with configuration verification and runtime access testing.

How insecure training patterns enter production code

Generative models often reproduce the patterns most visible in public repositories, including deprecated libraries, hardcoded secrets, and insecure dependency choices. If the training data contains legacy examples of weak hashing, permissive access patterns, or unverified packages, the model may treat those as normal implementation options. The danger is not that the model is malicious. It is that it lacks a security objective unless one is explicitly imposed during prompting, review, and validation.

Practical implication: embed security context before code is generated so insecure patterns are less likely to be accepted.

Threat narrative

Attacker objective: The attacker aims to turn functional but insecure production code into a path for unauthorised data access, credential theft, or account takeover.

1. Entry occurs when a developer accepts AI-generated code that appears functional but embeds insecure database queries, weak authentication logic, or unvetted dependencies.
2. Credential exposure or access failure follows when generated code or supporting infrastructure leaves tokens, secrets, or authorisation boundaries improperly protected.
3. Impact arrives when the application reaches production and attackers exploit the missing runtime controls to access data, bypass authentication, or abuse exposed credentials.

Breaches seen in the wild

• 12,000 secrets in LLM training data: Truffle Security found 11,908 live API keys and passwords hard-coded in web pages captured by Common Crawl, a dataset used to train LLMs.
• Firebase misconfiguration exposure 2024: Missing Firebase security rules on 916 websites exposed 125 million user records and 19.87 million plaintext passwords; a quarter were fixed.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI-generated code creates a validation gap, not just a code-quality gap. The important issue is that security teams can no longer assume successful compilation or passing tests means the control environment is safe. When authentication, database isolation, and secret handling are delegated to generated code, the real failure mode is missing verification of runtime policy. Practitioners should treat generated software as untrusted until its security properties are proven.

Runtime authorisation is the decisive control boundary. This article shows that access control failures often live outside the code the developer reviews, especially where row-level controls, cloud permissions, or service credentials determine who can reach data. That makes the identity layer part of the application security problem, not a separate concern. Teams need assurance that authorisation actually executes in production, not just in design diagrams.

Secrets leakage and insecure dependencies are symptoms of prompt-era engineering debt. AI systems frequently reproduce legacy patterns because they are statistically common, not because they are safe. That means insecure outputs can scale faster than traditional secure development practices can compensate. The practical conclusion is that security context must be injected earlier in the development flow, before the model emits code that later has to be rescued.

Functional speed without security evidence is an organisational risk multiplier. The article’s deeper message is that AI accelerates release cadence faster than most governance processes can adapt. The security programme therefore needs a new checkpoint: evidence that generated code resists abuse, not only evidence that it executes. Teams that cannot produce that evidence are shipping with unresolved assurance debt.

AI coding risk is now a governance issue for human and machine identity alike. When generated code handles API keys, service tokens, and delegated access, it affects how NHIs are created, scoped, and trusted. That means the security boundary extends into machine identity governance, especially where applications use tokens or service accounts to reach databases and cloud services. Practitioners should govern generated code as part of identity assurance, not as an isolated developer productivity problem.

From our research library:

• 13% of organisations suffered breaches of their AI models, according to IBM's Cost of a Data Breach 2025 report.
• Read next: AI Agent Identity Security Buyer's Guide

What this signals

AI-generated code security needs runtime proof, not output confidence. The programme-level question is whether teams can demonstrate that generated code enforces authorisation and secrets hygiene after deployment, because the model’s ability to produce working code says nothing about the strength of the surrounding control boundary. The operational shift is from code review as the primary gate to control verification as the primary gate.

Machine identity governance now intersects with application generation. Generated applications often rely on API keys, service tokens, and delegated access paths that are easy to create and difficult to govern later. That makes NHI controls relevant at design time, not only after a service account has already been deployed into production. Teams should assume the access layer is part of the build artefact and govern it accordingly.

Adversarial testing becomes the practical separator between safe velocity and unsafe velocity. If security checks only inspect whether the code compiles, the organisation will miss exactly the failures AI systems are most likely to create. The more quickly teams generate software, the more they need tests that try to break authorisation, data isolation, and credential handling before release.

For practitioners

• Embed security prompts before code generation Require threat-aware prompt templates that ask for input validation, least privilege, secret handling, and dependency review before the model writes code.
• Validate runtime authorisation, not just source code Test whether row-level security, token checks, and cloud permissions actually block unauthorised access after deployment, especially for data-heavy features.
• Add adversarial tests to AI-built features Use negative test cases for injection, access bypass, secret exposure, and unsafe dependency use so security failures appear before release.
• Track secret exposure across the delivery pipeline Scan generated code, build artefacts, and configuration outputs for API keys, tokens, and credentials that may have entered through AI-assisted development.
• Gate production release on security evidence Block release until scanning, runtime checks, and review evidence confirm that the generated application enforces the intended access boundaries.

Key takeaways

• AI-generated code can look correct while still missing the runtime controls that protect data, credentials, and access boundaries.
• Research cited in the article shows the gap is material, with developers often failing to follow secrets-management best practices and AI tools repeatedly producing insecure patterns.
• The practical response is to validate generated code with adversarial testing, runtime policy checks, and stricter governance over secrets and machine credentials.

Key terms

• Adversarial Testing: A testing approach that tries to break a policy by using hostile or unexpected inputs. For PBAC and AI access controls, that means probing for prompt injection, role crossover, leakage, and connector drift so the organisation can see whether the policy still holds under pressure.
• Runtime Authorisation: Runtime authorisation is the practice of deciding access while a task is in progress, rather than only at provisioning time. It matters for NHIs because credentials and entitlements can change risk mid-session, especially when automation or AI agents interact with sensitive systems.
• Secrets Exposure: Secrets exposure is the accidental or uncontrolled disclosure of credentials such as API keys, tokens, certificates, and service passwords. In NHI programs, it matters because a leaked secret often behaves like a live identity, creating immediate access risk until it is revoked or rotated.
• Machine Identity Governance: Machine Identity Governance is the discipline of controlling how non-human identities are created, used, monitored, and retired. It covers service accounts, API keys, certificates, tokens, workloads, and automation identities, with policies for ownership, lifecycle, least privilege, rotation, attestation, and auditability across cloud, application, and infrastructure environments.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management for teams that need to govern machine access paths created by modern development workflows. It helps security, IAM, and engineering leads connect identity controls to the broader release process.