AI application security is shifting toward agent and model governance

At a glance

What this is: This is an analysis of AI application security and its key claim that traditional AppSec is not enough for AI systems with model, data, and agent-specific attack surfaces.

Why it matters: It matters because IAM, PAM, and security teams increasingly need to govern AI systems as identity-bearing runtime actors, with access controls, monitoring, and lifecycle discipline that extend beyond classic code scanning.

By the numbers:

• According to a report by Microfocus, 61% of the applications tested were found to have at least one vulnerability that was not included in the OWASP Top 10 list.
• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.

👉 Read Akto's analysis of AI application security for safer AI systems

Context

AI application security extends AppSec into the model, data, and runtime layer, where prompt injection, model theft, data poisoning, and unsafe outputs create attack paths that standard code-centric controls do not reliably catch. The primary keyword here is AI application security, and the real governance question is how to secure AI-powered applications without treating them like ordinary software.

That distinction matters for identity programmes because AI systems increasingly act through APIs, tools, secrets, and delegated access. Where AI agents can select actions at runtime, the security problem becomes one of controlling identity, privilege, and trusted inputs across the full execution chain, not just scanning source code before release.

Key questions

Q: How should security teams implement AI evaluation in production workflows?

A: Security teams should treat AI evaluation as a continuous control, not a pre-launch checklist. Build representative datasets, define scoring criteria for the outcomes that matter, and rerun tests whenever prompts, models, or retrieval logic change. That creates evidence for regression detection and release decisions instead of relying on intuition.

Q: Why do AI systems create identity and access risk beyond traditional AppSec?

A: Because AI systems often act through delegated access. When a model can use tools, retrieve data, or trigger actions, it becomes a runtime decision-maker with privileges that can be misused through prompt injection, poisoned context, or overbroad permissions. That is an identity problem as much as a code problem.

Q: What breaks when AI outputs are validated without runtime monitoring?

A: Offline validation can confirm expected behaviour in test conditions, but it cannot detect live abuse after deployment. If prompts, retrieval content, or connected tools change, a model can still leak data or take unsafe actions. Without runtime monitoring, teams lose the visibility needed to distinguish normal variation from compromise.

Q: When should organisations treat an AI agent as a privileged system?

A: Organisations should treat an AI agent as privileged whenever it can reach production data, administrative tools, or sensitive workflows without direct human approval for each step. At that point, the agent is no longer a passive automation helper. It becomes a governed identity whose permissions, logs, and exceptions need the same scrutiny as other high-risk access.

Technical breakdown

Why AI application security is different from traditional AppSec

Traditional AppSec focuses on source code, dependencies, and runtime flaws such as injection or authentication mistakes. AI application security has a wider attack surface because models can be manipulated through inputs, training data, tool use, and output handling. A secure model can still be exploited if prompts are steered, context is poisoned, or the system is given excessive access to secrets and APIs. The boundary shifts from code correctness to trust in behaviour, data lineage, and runtime authority.

Practical implication: teams need controls that evaluate model inputs, outputs, and delegated access, not just code vulnerabilities.

How prompt injection and model abuse create identity and access risk

Prompt injection is a way of influencing an AI system’s decisions by embedding malicious instructions in user input, retrieved content, or connected tools. Model abuse extends that pattern to misuse of capabilities such as data access, function calls, and action execution. In practice, the model becomes a decision point that can be tricked into using identities, tokens, or workflow permissions in ways the owner did not intend. That is why AI security and identity governance now intersect directly.

Practical implication: restrict tool scope, bind actions to task context, and treat model permissions as governed entitlements.

Why runtime monitoring and model validation need to operate together

AI systems are probabilistic, so a model that passes pre-release checks can still fail in production when context, prompts, or data drift. Model validation tests expected behaviour, while runtime monitoring looks for anomalous prompts, unusual outputs, or suspicious access patterns. The important point is that neither layer is sufficient alone. Validation without telemetry misses live abuse, and telemetry without a baseline cannot tell normal variation from compromise.

Practical implication: pair offline testing with live detection so AI controls work before, during, and after deployment.

Threat narrative

Attacker objective: The attacker aims to manipulate AI behaviour, extract sensitive information, or abuse connected systems through the model’s trusted runtime position.

1. Entry occurs when malicious prompts, poisoned content, or compromised dependencies reach the AI application through user interactions, retrieval channels, or training inputs.
2. Escalation follows when the model is induced to reveal sensitive data, call tools outside intended scope, or process hidden instructions as trusted context.
3. Impact appears as data leakage, model theft, unsafe actions, or downstream misuse of delegated access across the AI application stack.

NHI Mgmt Group analysis

AI application security is becoming an identity governance problem, not just an AppSec problem. Once a model can call tools, query data, or trigger workflows, it inherits a form of runtime authority that must be governed like any other privileged system. The practical issue is not only whether the model is secure, but whether its access is bounded, reviewable, and revocable in the same way as sensitive service access.

Model behaviour is now part of the attack surface. AI systems do not fail only through code defects. They can be steered through prompts, poisoned inputs, and overbroad tool permissions into leaking secrets or taking unintended actions. That means security teams should stop treating output quality as a separate concern from access control, because in AI systems, behaviour and privilege are tightly linked.

AI governance debt emerges when teams add models faster than they add controls. A useful named concept here is AI governance debt: the backlog of missing controls, baselines, and accountability that accumulates when AI features go live before identity, monitoring, and validation are defined. The longer that debt persists, the harder it becomes to distinguish legitimate model autonomy from misuse. Practitioners should treat every new AI workflow as a governance design event.

Secrets and delegated access remain the most fragile layer in AI application stacks. The article’s emphasis on monitoring, RBAC, and MFA is directionally right, but the larger point is that AI systems frequently inherit credentials they do not need persistently. When those credentials are shared across pipelines, apps, and agents, compromise becomes both easier and harder to trace. Teams need to narrow delegation and make every high-value permission explicit.

What this signals

AI application security programmes are likely to converge with identity governance as soon as models begin using real credentials, not mock ones. That shift means AI workflows need ownership, offboarding, and entitlement review in the same operational model used for human and non-human identities.

Model authority sprawl: when AI systems inherit access through assistants, pipelines, and plugins, entitlement creep becomes harder to spot than in conventional software. The control signal to watch is not only failed detections, but also the number of AI-linked identities that can reach sensitive data or production tools.

The security signal for practitioners is clear: if your AI stack depends on the same shared secrets and static permissions that already prolong leaked-secret remediation, the exposure window will remain too wide. Tighter credential lifecycle control and better runtime logging should be treated as baseline requirements, not future enhancements.

For practitioners

• Bound AI tool permissions to task scope Define the minimum set of tools, APIs, and datasets each AI workflow can touch, then revoke anything that is not required for the task. Use short-lived credentials where possible and separate read from write access for high-impact actions.
• Add prompt and output controls to the SDLC Treat prompt injection, unsafe output handling, and retrieval poisoning as design-time risks. Build test cases into release pipelines that verify the model ignores hidden instructions, resists malicious context, and does not expose secrets through responses.
• Instrument runtime monitoring for AI-specific abuse Log model calls, tool invocations, and anomalous response patterns into SIEM workflows so suspicious behaviour can be triaged quickly. Focus on identity-linked signals such as unexpected token use, new integrations, and unusual privilege paths.
• Review agent and model credentials as a separate control set Inventory credentials, keys, certificates, and service accounts used by AI systems and rotate or isolate them on a tighter schedule than general application secrets. Model-facing identities should have explicit ownership and offboarding logic.
• Validate AI controls continuously after launch Run periodic adversarial testing and revalidation whenever prompts, models, retrieval sources, or integrations change. AI security is operational, not a one-time review, because behaviour shifts as dependencies and context shift.

Key takeaways

• AI application security now spans prompts, models, data, tools, and delegated access, which makes it broader than traditional AppSec.
• The central governance issue is not only whether AI behaves safely, but whether its runtime authority is bounded and reviewable.
• Teams should pair model validation with runtime monitoring and treat AI credentials as a distinct privileged control set.

Key terms

• AI Application Security: AI application security is the practice of protecting AI-powered software from attacks that target prompts, models, data pipelines, tools, and outputs. It extends application security into behavioural and runtime controls because the model can be manipulated even when the underlying code is not obviously flawed.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• AI Governance: AI governance is the set of controls used to discover, classify, approve, restrict, monitor, and revoke AI-enabled access. It connects identity, data, and policy so organisations can manage what AI can reach, what it can share, and when it should be stopped.
• Delegated Access: Delegated access is permission granted to one identity to act on behalf of another user, service, or system. In NHI environments, this usually appears in OAuth-connected apps and automation tooling. It is powerful, but it must be tightly scoped and reviewed because it can persist long after the original business need ends.

What's in the full article

Akto's full blog covers the operational detail this post intentionally leaves for the source:

• How Akto positions AI firewalls, runtime monitoring, and SIEM integration inside the AI application security lifecycle
• The article's side-by-side comparison of traditional AppSec and AI AppSec across attack surface, tools, and assessment methods
• The practical implementation framing for AI security in DevSecOps pipelines, including model validation and continuous monitoring
• Akto's own view of how its agentic AI suite fits into API security and AI application protection

👉 Akto's full post covers the AI AppSec versus traditional AppSec comparison and implementation flow in more detail.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and identity lifecycle control. It gives practitioners a structured way to connect identity governance to the broader security programme they operate.