AppSec-to-AI security needs new controls for AI systems

At a glance

What this is: A Protect AI blog post explains how application security thinking carries into AI security, while showing where AI systems introduce model, data, and runtime risks that AppSec alone does not cover.

Why it matters: IAM, NHI, and security teams need to treat AI systems as governed access surfaces, because model capabilities, data access, and runtime behaviour create new control points across identity programmes.

👉 Read Protect AI's security engineer perspective on AppSec to AI security

Context

AI security is not just an extension of application security, because the control surface expands from code and infrastructure to models, training data, prompts, and runtime behaviour. That matters for identity governance because access is now being exercised by systems that may read, transform, or expose sensitive information in ways traditional AppSec workflows do not model well.

The practical gap is governance, not awareness. Most security programmes already understand input validation, authorization, and supply chain hygiene, but AI systems introduce new decision points around model provenance, fine-tuning access, and behavioural monitoring that need identity-aware controls across NHI, human, and autonomous workflows.

Key questions

Q: How should security teams govern AI systems that can access sensitive data?

A: Security teams should treat AI systems as governed access paths, not just applications. That means separating permissions for prompts, retrieval sources, training data, and outputs, then assigning ownership for each path. The key is to make model behaviour traceable so data access, content generation, and operational change are all reviewable within the identity programme.

Q: Why do traditional AppSec controls fall short for AI applications?

A: Traditional AppSec controls fall short because they focus on code integrity and request validation, while AI systems also need controls for model behaviour, training data lineage, and output safety. A model can be manipulated without a classic exploit, so teams need governance that covers both the application and the decision-making layer.

Q: What do security teams get wrong about prompt injection?

A: Teams often treat prompt injection as a text filtering issue, but it is really a boundary-crossing issue. The risk is that a hostile prompt changes how the model interprets instructions and what it can expose or trigger next. Defences need to cover context, tool access, and runtime monitoring, not just input sanitisation.

Q: How can organisations know whether AI monitoring is actually working?

A: AI monitoring is working when it detects abuse patterns before they become harmful outputs or data leakage events. Useful signals include prompt-injection attempts, jailbreaks, sensitive data disclosure, and anomalous tool use. If the programme only reports uptime or request volume, it is measuring activity, not security.

Technical breakdown

Prompt injection vs classic input validation

Prompt injection is the AI analogue of malicious input, but the failure mode is broader than a single payload. In a normal application, input validation blocks unsafe characters or malformed requests. In an AI system, user text can redirect the model’s behaviour, override intended instructions, or cause disclosure across tool-connected workflows. The issue is not just bad text, but the model’s susceptibility to instruction hierarchy confusion and context contamination. That makes it a runtime governance problem as much as an application filtering problem, especially when the model can call tools or handle sensitive data.

Practical implication: validate prompts, constrain tool use, and monitor for instruction-overriding patterns at runtime.

Model governance, training data, and authorization boundaries

AI systems create separate authorization boundaries for model features, training data access, and downstream outputs. That is different from traditional app roles, where permissions usually map to data objects or application functions. When a model is trained, tuned, or given retrieval access, the security question becomes who can influence behaviour, who can inspect lineage, and who can expose embedded knowledge. Model inventories, versioning, and clear guardrails are governance controls because they define what the system is allowed to know and do. Without them, security teams lose traceability across both data and behaviour.

Practical implication: inventory models and training data, then bind each to explicit ownership, access, and approval controls.

Runtime detection for AI-specific abuse

Traditional monitoring looks for anomalous logins, traffic spikes, and known exploit patterns. AI monitoring has to add prompt injection attempts, jailbreaks, PII leakage, code leakage, and harmful output generation. That changes observability from passive log review to behaviour-aware enforcement. The point is not simply to detect that the model was used, but to identify when it crossed an intended boundary. This is where MLSecOps-style monitoring fits: it bridges security, data science, and operations so that model behaviour can be evaluated continuously, not only at release time.

Practical implication: add AI-specific detections for prompt abuse, leakage, and harmful output to the operational security stack.

NHI Mgmt Group analysis

AppSec is necessary for AI security, but it is not sufficient. The article is right that familiar controls still matter, especially validation, authorization, and supply chain review. But AI systems add a behavioural layer that traditional AppSec was never designed to govern, because the model can reinterpret input rather than merely process it. The implication is that teams should stop treating AI as a normal application with a different interface and start treating it as a governed decision surface.

Prompt injection is a control-boundary problem, not just a content problem. In conventional software, input validation assumes the application will execute only the logic developers wrote. In AI systems, hostile prompts can alter the effective instruction set and shift the model outside intended behaviour. That changes the risk model for every downstream tool the model can reach. Practitioners should recognise that the security boundary now includes the prompt, the model context, and any connected actions.

Model governance becomes an identity problem once access is extended to data, weights, and tools. Model inventories, version control, and training-data lineage are not administrative extras. They are the controls that determine who can shape behaviour, what the model can learn, and where trust begins and ends. This is where NHI thinking becomes useful: every machine-facing access path needs ownership, scope, and traceability, or governance becomes guesswork.

Runtime observability must move from detection of usage to detection of boundary crossing. The article’s monitoring section points in the right direction, but the deeper lesson is that AI systems fail in ways ordinary logs do not capture. A model can leak data, reveal hidden context, or generate unsafe outputs without any classic intrusion event. The named concept here is behavioural boundary drift: a model operating outside the security assumptions under which it was deployed. Practitioners need controls that detect that drift early enough to matter.

From our research:

• 64% of valid secrets leaked in 2022 are still valid and exploitable today, according to The State of Secrets Sprawl 2026.
• 24,008 unique secrets were exposed in MCP configuration files in 2025 alone, showing how quickly new AI toolchains create identity exposure paths.
• For a broader NHI lens, the 52 NHI breaches Report shows how exposed credentials and weak lifecycle controls turn detection gaps into lasting compromise.

What this signals

Behavioural boundary drift: AI security programmes need to watch for models that move beyond intended context, tool use, or disclosure boundaries. Once a model can retrieve data, call tools, or shape output dynamically, the control question becomes whether the system is still acting inside the authority it was given.

Secrets and model access should be handled together in operational planning, because exposed credentials, retrieval misconfigurations, and weak offboarding all expand the same attack surface. The lesson for practitioners is to align model governance with NHI lifecycle discipline, not to bolt AI monitoring onto an unrelated AppSec stack.

For practitioners

• Define AI-specific access boundaries Map which users, service accounts, and pipelines can influence prompts, training data, retrieval sources, and model outputs. Separate those permissions so model access is not bundled with broad platform access.
• Add prompt abuse detections Instrument logging for prompt injection attempts, jailbreak patterns, and instruction-overriding content, then route those signals into the same triage process used for other high-risk security events.
• Inventory models and training data Track each model, version, fine-tuning source, and retrieval dependency with an owner and review date so you can prove what the system was exposed to and who approved it.
• Test for leakage before release Use red-team style testing to check whether the model reveals PII, code, or internal context under adversarial prompts, then block deployment until the leakage paths are understood.

Key takeaways

• AI security extends AppSec, but only if teams govern models, data lineage, and runtime behaviour as first-class security concerns.
• Prompt injection and leakage risks show that static controls alone cannot describe how AI systems fail once they start interacting with tools and sensitive data.
• Identity programmes should treat AI access paths, secrets, and output monitoring as one governance problem rather than three separate projects.

Key terms

• Prompt Injection: Prompt injection is a technique that manipulates an AI model through crafted input so it follows attacker intent instead of expected instructions. In security terms, it is a boundary failure between trusted system guidance and untrusted user content, especially dangerous when the model can access tools or sensitive data.
• Model Governance: Model governance is the set of controls that define who can build, tune, approve, deploy, and monitor an AI model. It includes ownership, version control, lineage, access boundaries, and documented limitations so the organisation can explain what the model is allowed to do and what it must not do.
• Runtime Threat Detection: Runtime threat detection is monitoring that looks for abuse while an AI system is operating, not only during testing or release. It focuses on prompt attacks, leakage signals, abnormal tool use, and unsafe outputs so security teams can interrupt harmful behaviour while the model is still active.
• Behavioral Boundary Drift: Behavioral boundary drift is the condition where an AI system starts operating outside the security assumptions that were set for it at deployment. It can appear as unexpected disclosure, tool misuse, or context expansion, and it is often missed by controls that only measure request volume or system uptime.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.

This post draws on content published by Protect AI: Security Spotlight, AppSec to AI, a security engineer's journey. Read the original.