AppSec-to-AI security needs new controls for AI systems

At a glance

What this is: This is a practitioner view of how AppSec thinking maps to AI security, with the key finding that conventional application controls do not fully cover prompt, model, and data risks.

Why it matters: It matters because IAM, security architecture, and governance teams now have to treat AI systems as a wider control surface where access, data protection, and runtime oversight all intersect.

Context

AI security extends application security into a broader control plane that includes models, training data, and runtime behaviour. Traditional code scanning and perimeter testing still help, but they do not address prompt injection, model poisoning, or the governance needed for model inventories and access boundaries.

For identity and governance teams, the important shift is that AI systems create security decisions around who can access training data, which model features can be used, and how model behaviour is monitored after deployment. That makes AI security a control problem, not just a software testing problem.

Key questions

Q: How should security teams extend AppSec controls to AI systems?

A: Security teams should keep input validation, authentication, and least privilege, but extend them to prompts, model access, training data, and runtime monitoring. AI systems create new trust boundaries, so governance must cover model assets and behaviour, not only code and infrastructure. The goal is to make AI part of the existing assurance model without pretending it is the same control surface.

Q: Why do traditional application controls miss the biggest AI security risks?

A: Traditional controls focus on code, dependencies, and perimeter testing, while AI risk also includes model poisoning, prompt injection, training data leakage, and output abuse. Those behaviours can alter model decisions without breaking the underlying application. The gap is not that AppSec is irrelevant, but that it stops at the wrong boundary for AI systems.

Q: What are the signs that AI penetration testing is not giving enough coverage?

A: Coverage is likely too narrow when testing only runs on snapshots, checks a limited set of endpoints, or fails to adapt after an error or unexpected response. Weak coverage also shows up when teams still depend on manual retesting to confirm obvious issues. Effective tools should explore many paths, keep context, and continue testing as the environment changes.

Q: What is the difference between AppSec and AI security governance?

A: AppSec secures software applications through code, runtime, and infrastructure controls. AI security governance adds oversight of model provenance, training data, behavioural limits, and post-deployment monitoring. The difference is that AI can be influenced by data and prompts after deployment, so governance must follow the model throughout its lifecycle.

Technical breakdown

Prompt injection changes the trust model for AI inputs

Prompt injection is the AI analogue of malicious input, but the failure mode is broader than classic injection because the model may follow instructions that were never intended to be trusted. In application security, validation keeps hostile input from altering program flow. In AI systems, the model can be steered by natural language, hidden instructions, or context poisoning even when the surrounding application is technically intact. That means the defensive boundary has to include prompt handling, system prompt protection, and output validation, not just web application filters.

Practical implication: treat prompts and retrieved context as governed inputs, not harmless text.

Model poisoning and training data exposure are supply chain problems

AI supply chain risk includes the provenance of pre-trained models, the quality of training data, and the security of fine-tuning artefacts. Model poisoning occurs when adversaries tamper with data used to shape model behaviour, while training data leakage can expose sensitive records embedded in the model's learning process. These are not ordinary dependency issues because the compromise can survive deployment and influence every response the model produces. Governance has to extend to lineage, versioning, and approved source control for model assets and training corpora.

Practical implication: inventory model assets and secure the full training and fine-tuning lineage.

Runtime observability is now part of the control stack

AI systems need monitoring for prompt injection attempts, jailbreaks, PII leakage, harmful output, and anomalous model behaviour. Traditional SIEM-style logging is useful, but AI telemetry has to capture model inputs, outputs, guardrail events, and evaluation results so that security and MLOps teams can see when controls are failing. This is where MLSecOps becomes relevant: it joins detection, validation, and operational response around model behaviour instead of only infrastructure signals. Without that visibility, security teams are blind to misuse that occurs after deployment.

Practical implication: build monitoring for model inputs, outputs, and guardrail failures into production operations.

Breaches seen in the wild

• Shai Hulud npm malware campaign: Shai Hulud campaign: npm malware exposed secrets on GitHub.
• reviewdog Action compromise 2025: A stolen maintainer token poisoned reviewdog/action-setup, leaking CI secrets including the tj-actions bot token used in the next attack.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AppSec principles still apply, but they no longer define the boundary of control. AI systems inherit input validation, authentication, and least privilege, yet the protected object is no longer just code and data. Models can be manipulated through prompts, training data, and output channels, so the control plane expands beyond the application perimeter. The practitioner conclusion is simple: AI security is AppSec plus model and data governance, not AppSec renamed.

Model governance is the missing discipline in most AI programmes. A model inventory, version control, risk classification, and documented limitations are not optional metadata; they are the operational record of what the organisation has allowed into production. When those controls are absent, teams cannot answer basic questions about provenance, accountability, or rollback. The practitioner conclusion is that AI systems need explicit governance artefacts before they need more testing.

Runtime observability is the decisive control gap in AI operations. Security teams are used to logs, alerts, and detection rules, but AI systems require telemetry on model behaviour, prompt abuse, and harmful output. That makes post-deployment monitoring a core security function, not a nice-to-have operational layer. The practitioner conclusion is that detection must move from infrastructure events to model-level signals.

AI security programmes fail when they treat training data as a passive input store. Training data is part of the security boundary because it shapes the behaviour of the model and can encode sensitive information that later reappears in responses. That turns data provenance into a security control, not just a data science concern. The practitioner conclusion is that governance over model training data belongs in the security programme, not only in ML engineering.

Secure AI development should be assessed as a lifecycle, not a collection of point controls. Security requirements, threat modelling, validation, and monitoring only work together when they are designed as a continuous process from model selection through deployment and operations. That mirrors mature software assurance, but the artefacts are different. The practitioner conclusion is to govern AI with a lifecycle model that includes models, data, and runtime behaviour.

From our research library:

• 71% of organizations use third-party APIs, according to Gartner’s 2024 data.
• Read next: Agentic AI Security Guide

What this signals

Model governance is now a security control: once AI systems move into production, the question is no longer only whether the code is secure. Security leaders need a model inventory, approved training data lineage, and runtime oversight so they can see what the system learned, what it can access, and when its behaviour drifts.

AI programmes that stop at application testing will miss the boundary where prompts, model weights, and training data become part of the attack surface. That means review cycles, logging, and control ownership all need to expand beyond traditional AppSec.

Organisations should treat AI-specific telemetry as a first-class control, not an afterthought. Detection has to include prompt abuse, jailbreaks, and harmful output because those are the operational symptoms of governance failure in production.

For practitioners

• Define AI security requirements separately from AppSec requirements Map prompt handling, model access, training data protection, and runtime monitoring into their own control set so they are not diluted inside generic application security standards.
• Inventory models, prompts, and training data as governed assets Track model versions, approved data sources, fine-tuning artefacts, and known limitations so security teams can answer what is deployed and what influenced it.
• Add AI-specific testing to your assurance workflow Use model scanning, automated red teaming, and runtime abuse testing alongside SAST, SCA, and DAST to cover attack paths that traditional tooling misses.
• Build production telemetry for model behaviour Log prompt injection attempts, jailbreaks, abnormal outputs, and guardrail failures so detection and response can operate on AI-specific signals.
• Establish ownership for AI governance artefacts Assign clear accountability for model inventories, access to training data, and control exceptions so security, data science, and engineering do not each assume another team owns them.

Key takeaways

• Traditional AppSec remains relevant for AI systems, but it does not cover the full control surface once prompts, models, and training data become part of the risk model.
• AI security failures often arise after deployment, which means model governance and runtime monitoring matter as much as code testing.
• Teams that want usable AI controls need separate governance for model inventory, data lineage, and behavioural monitoring, not just stronger web application security.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Model Poisoning: Model poisoning is the deliberate corruption of training or ingestion data so an AI system learns or behaves incorrectly. The attack targets the trust chain before or during model training, which means the resulting failure can persist into production outputs, decisions, and automated actions.
• Training Data Lineage: A record of where training data came from, how it changed, and whether it met quality or compliance requirements before model use. It matters because biased, sensitive, or unlawfully sourced data can shape AI behaviour long after training is complete, especially in regulated workflows.
• Trace Observability: Trace observability is the ability to inspect a workflow step by step, including inputs, decisions, retries, and outputs. For AI agents, traces show where a loop is wasting tokens, hitting permission barriers, or failing to converge. That visibility supports debugging, cost control, and post-run review.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.