Generative AI security is exposing a new AppSec attack surface

At a glance

What this is: This is an independent analysis of how generative AI is changing application security, with the key finding that risk now starts in prompts, models, and training data before code reaches traditional controls.

Why it matters: It matters because AppSec, IAM, and platform teams now have to govern AI-assisted development, model-driven behaviour, and untracked integrations without assuming that legacy scanning or review workflows can see the full attack surface.

By the numbers:

• According to Palo Alto Networks, there was an 890% surge in generative AI traffic in 2024, showing how quickly AI-assisted development has become embedded in the software ecosystem.
• GenAI data loss incidents increased by 250% in early 2025 and now account for 14% of all data security incidents across SaaS traffic.
• A joint study by Checkmarx and Jit.io found that 80% of organisations using AI coding assistants ship vulnerable code, including missing authentication checks and unsafe defaults.

👉 Read Apiiro's analysis of generative AI security and the AppSec blind spot

Context

Generative AI security is now an application security problem, a software supply chain problem, and increasingly an identity and access problem. The prompt, the model, the plugin, and the runtime service can each become an attack path, which means the old assumption that risk begins at commit time no longer holds.

Traditional AppSec programmes were built around human-written code, predictable review cycles, and tools that can reason from source to build to deploy. AI-assisted development breaks that sequencing by introducing opaque logic, untracked dependencies, and behaviour that may only become visible once the application is already processing sensitive data or calling live services.

Key questions

Q: How should security teams govern AI experimentation without slowing delivery?

A: Use lane-based governance. Low-risk experiments belong in isolated sandboxes with minimal guardrails, while managed and critical workloads should inherit progressively stronger controls. The goal is to make experimentation safe by default, not to force every AI use case through production-grade approval gates before value is proven.

Q: Why do AI coding assistants still produce insecure code?

A: They learn patterns from large code corpora that contain both secure and insecure examples, and they do not reliably reason about all dataflow or trust-boundary issues. Where a task requires context-sensitive security judgment, the model can produce plausible code that still mishandles input, encoding, or privilege. Better reasoning helps, but it does not replace analysis.

Q: What breaks when organisations cannot inventory their AI credentials?

A: Rotation, recertification, and offboarding all break down when the inventory is incomplete. Teams cannot prove which keys are active, cannot identify which owners should review them, and cannot confidently retire credentials that may already be obsolete. The result is governance theatre: policies exist, but the evidence needed to enforce them does not.

Q: How can security teams tell whether AI-generated code is actually safe?

A: They should look for security regression evidence, not just test pass rates. A codebase is safer only when the agent’s output survives threat-informed checks, review of privileged paths, and validation against the vulnerability classes most likely to recur in that repository.

Technical breakdown

Why prompts and model inputs now count as an attack surface

In generative AI systems, a prompt is not just user input. It can shape code generation, tool invocation, and downstream system behaviour, which makes it a security boundary rather than a convenience layer. Prompt injection works because instructions are treated as text unless the application explicitly constrains them, so malicious input can override intended behaviour, trigger data exposure, or manipulate agent actions. This changes the meaning of input validation. It is no longer enough to sanitize forms or APIs after code is written, because the earliest unsafe instruction may exist before any code commit. Secure design now has to account for prompt context, tool permissions, and instruction hierarchy.

Practical implication: Treat prompts, templates, and tool instructions as governed inputs with validation, filtering, and scope limits before they reach model or agent execution.

How AI-generated code creates a comprehension gap in SDLC controls

AI coding assistants increase throughput, but they also reduce the reviewer’s ability to understand intent. That creates a comprehension gap, where code may compile and pass tests while still hiding unsafe control flow, unverified dependencies, or weak authentication logic. Legacy SAST, DAST, and SCA tools were never designed to resolve this gap at scale because they look for known signatures rather than architectural intent. When code generation accelerates faster than review capacity, security teams spend more time triaging noisy findings and less time understanding whether a change is actually safe. The real issue is not only vulnerability density, but loss of human comprehension across the codebase.

Practical implication: Add context-aware review gates that link AI-generated changes to ownership, data flow, and runtime exposure rather than relying on static scan results alone.

Why Shadow GenAI turns governance into an inventory problem

Shadow GenAI describes unsanctioned AI frameworks, plugins, models, and dependencies that enter applications without formal review. This is structurally similar to shadow IT, but more difficult to govern because AI components often connect to external APIs, process sensitive data, or persist in supply chains through indirect updates. Without a live inventory, teams cannot tell which models are in use, what data they touch, or which services they can reach. That makes risk assessment speculative. In practice, governance fails first at visibility, then at policy enforcement, and finally at incident response because teams cannot prove where AI components exist or how they behave.

Practical implication: Build an authoritative inventory of AI frameworks, models, and plugins and tie it to approved data access, runtime telemetry, and ownership records.

Threat narrative

Attacker objective: The attacker aims to turn AI-assisted development or agentic workflows into a trusted path for data exposure, code compromise, or unauthorised system action.

1. Entry begins when an attacker manipulates prompts, poisoned training data, or an unreviewed AI plugin to influence model behaviour or execution paths.
2. Escalation occurs when the model or AI agent inherits trusted context, over-permissioned tooling, or unsafe defaults that let it access systems it should not reach.
3. Impact follows when the manipulated AI component leaks sensitive data, issues unauthorised actions, or introduces insecure code into production workflows.

NHI Mgmt Group analysis

Generative AI has turned application security into a prompt-to-runtime governance problem. The security boundary has shifted upstream from commit and build into prompt construction, model configuration, and tool delegation. That means teams can no longer treat AI-assisted code as a downstream scanning issue. The relevant control model now spans architecture, policy, and runtime telemetry, not just source analysis. Practitioners should govern AI systems as part of the SDLC itself, not as a separate innovation layer.

Shadow GenAI is the new shadow IT, but with higher consequence. Unapproved models, plugins, and embedded AI components create an inventory gap that breaks both risk assessment and accountability. When teams cannot enumerate which AI services touch sensitive data, they cannot apply consistent access control or incident response. In identity terms, this is an access governance failure as much as an application one. Practitioners should treat AI component discovery as a control requirement, not an optional hygiene exercise.

AI-generated code exposes a new comprehension debt in secure development. The problem is not simply that more code is produced. It is that less of it is understood well enough for peer review, threat modelling, or policy enforcement to remain reliable. That weakens assumptions inside application risk management, CI/CD controls, and compliance attestations. The security team must now ask whether the organisation can explain the behaviour of the code it ships. Practitioners should make explainability a review criterion for AI-assisted changes.

Identity governance still matters because AI systems are becoming active actors in the application stack. As AI assistants and agents gain access to repos, APIs, and operational tools, their permissions start to resemble machine identities that need lifecycle control, scoped access, and auditability. This is where AppSec meets IAM and NHI governance. The organisation that can define who or what authorised the model, what it could access, and when that access should expire will have the stronger control position. Practitioners should extend identity governance to AI-enabled software workflows immediately.

Prompt-to-runtime traceability is the named control gap this topic exposes. The decisive failure is the inability to connect an AI input, the resulting code or action, and the live system impact in one traceable chain. Without that linkage, security teams cannot separate harmless experimentation from material exposure. This is where context-aware platforms, runtime observability, and change ownership need to converge. Practitioners should build controls that make AI behaviour attributable across the full lifecycle.

What this signals

Prompt-to-runtime traceability will become a baseline expectation for teams using AI in software delivery. The organisations that can connect prompts, generated code, deployed services, and runtime behaviour will be able to investigate faster and defend their decisions. That is the practical shape of AI security posture management in application programmes.

Identity teams should expect AI assistants and agents to be managed more like machine identities than like development tools. Once a system can call APIs, access repositories, or act on operational data, lifecycle, privilege, and audit controls matter as much as model quality. The governance model has to follow the capability, not the label.

The next control gap is not simply whether AI is approved, but whether the organisation can prove what it touched and why. That pushes security, AppSec, and platform teams toward continuous inventory, contextual policy, and tighter links between code changes and runtime evidence.

For practitioners

• Inventory all AI-assisted development paths Map every approved and unapproved model, coding assistant, plugin, and embedded framework to the repos and services it touches. Include hidden integrations introduced through dependency updates and developer-side experimentation, then tie each one to an owner and data classification.
• Add AI-specific review gates to SDLC workflows Require extra review when code is generated by assistants, when prompts can influence tool calls, or when a change introduces a model endpoint, external API, or unvetted package. Use these gates to force human review of intent, not only syntax or test coverage.
• Link source changes to runtime behaviour Correlate commits, model versions, and AI tool activity with runtime telemetry so teams can see which AI-generated change affected which asset or data flow. That traceability shortens triage and makes it easier to prove whether a suspicious action originated in code or model behaviour.
• Govern AI access like machine identity Assign least privilege to assistants and agents that can access repositories, APIs, or infrastructure. Enforce scoped permissions, short-lived credentials where possible, and audit trails that show who approved the access and when it should be revoked.
• Test for prompt injection and unsafe output handling continuously Run adversarial tests against prompts, tool instructions, and downstream output handling in the same cadence as application security testing. Include cases where model output becomes executable code, markup, or an API instruction, because that is where runtime exposure often begins.

Key takeaways

• Generative AI security is no longer an emerging side issue. It is now a core application security and governance problem because risk begins in prompts, models, and untracked components.
• Legacy scanners and review workflows cannot fully see AI-driven risk because the most important failures are contextual, behavioural, and runtime-dependent.
• Practitioners need inventory, traceability, and identity-style controls for AI-enabled workflows if they want to keep innovation moving without losing governance.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Shadow GenAI: Shadow GenAI refers to unsanctioned AI tools, models, plugins, and dependencies that appear in software environments without formal approval or visibility. It creates governance blind spots because teams cannot reliably classify data access, ownership, or exposure when AI components are introduced informally.
• Comprehension Gap: A comprehension gap is the point at which code can be generated faster than humans can understand it well enough to review, test, and govern it confidently. It weakens secure development because correctness, intent, and downstream behaviour become harder to verify than in traditional hand-written code.
• Prompt-to-Runtime Traceability: Prompt-to-runtime traceability is the ability to connect an AI input, the resulting code or action, and the live system effect in one audit chain. It is essential for investigating failures, proving accountability, and separating harmless experimentation from material security exposure.

What's in the full article

Apiiro's full article covers the operational detail this post intentionally leaves for the source:

• Concrete examples of how AI-generated code bypasses conventional AppSec review paths
• Detailed discussion of prompt injection, data poisoning, and insecure output handling in development workflows
• Practical workflow patterns for runtime monitoring and code-to-runtime correlation
• Implementation detail on AI-SPM style visibility and validation across models, code, and data

👉 Apiiro's full article covers the prompt, model, and runtime risks in more operational detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security and identity practitioners extend lifecycle control and access discipline into AI-enabled environments.