AI application security is shifting toward agent and model governance

At a glance

What this is: Akto argues that AI application security is moving beyond traditional AppSec to cover model governance, agent behaviour, and AI-specific attack paths such as prompt injection, data poisoning, and model theft.

Why it matters: IAM and security teams need to treat AI systems as governed assets with distinct control requirements, especially where model access, agent actions, and data pipelines create new abuse paths.

By the numbers:

• 61% of the applications tested were found to have at least one vulnerability that was not included in the OWASP Top 10 list, according to Microfocus research cited by Akto.

Context

AI application security extends application protection into the parts of AI systems that traditional AppSec does not model well: the model, the data pipeline, the agent layer, and the runtime behaviour those components produce. The article's core point is that AI systems create attack surfaces that are not solved by code scanning alone, especially when outputs, prompts, and external tool use all influence the security boundary.

That creates an identity and governance problem as much as a detection problem. When AI agents can act, call tools, or consume sensitive data, teams need controls for who or what is authorised to do that, under what conditions, and with what traceability. In that sense, the article is really about governing AI behaviour across the lifecycle rather than adding another point control.

Key questions

Q: How should security teams govern AI models that can call tools and access data?

A: Security teams should govern AI models as non-human identities with named owners, limited scope, short-lived credentials, and continuous authorization. The critical shift is to treat every tool call, data read, and update path as a privileged action that can be logged, revalidated, and revoked. Without that discipline, model risk becomes identity risk.

Q: Why do GenAI apps and agents need controls beyond AppSec and cloud security?

A: Because the AI system turns language, files, retrieved context, and tool access into behaviour. AppSec and cloud security still matter, but they do not test prompt injection, unsafe outputs, or whether an agent used authorised access for an unauthorised purpose. AI-specific controls close that gap at the interaction layer.

Q: What are the signs that an AI agent is being manipulated into unauthorized actions?

A: Warning signs include unexpected data retrieval, unusual tool calls, changes in agent decisions after hostile or extreme inputs, and outputs that reflect hidden instructions rather than the user’s request. In multi-agent environments, another signal is inconsistent behavior between agents, especially when one agent’s response appears to override normal guardrails or access boundaries.

Q: When should organisations prioritise AI governance over point-in-time testing?

A: They should prioritise governance as soon as models or agents begin influencing decisions, accessing sensitive data, or invoking tools. Point-in-time testing can validate a release, but it cannot control what changes during runtime, after model updates, or when new inputs reshape system behaviour.

Technical breakdown

Why traditional AppSec misses model and agent risk

Traditional AppSec is optimised for source code, APIs, dependencies, and known vulnerability classes such as injection or authentication flaws. AI applications add probabilistic behaviour, model outputs that change with context, and hidden dependencies in training data, prompts, and orchestration layers. That means a scanner can confirm that the code is syntactically sound while the model still remains vulnerable to prompt injection, data poisoning, model extraction, or unsafe tool use. The control problem shifts from static code defects to runtime governance of how the AI system reasons, responds, and interacts with other systems.

Practical implication: Practitioners should extend security testing beyond code to include model behaviour, prompt handling, and tool-use boundaries.

How data pipelines and model artefacts expand the attack surface

AI systems are only as trustworthy as the data and artefacts they consume. Poisoned training data, tampered model files, biased datasets, and unsafe fine-tuning inputs can all alter model behaviour without changing the surrounding application code. Unlike a conventional application bug, these issues can persist across deployments and be difficult to patch once embedded in the model or its surrounding pipeline. This is why AI security increasingly includes validation of datasets, model integrity checks, and monitoring of inference inputs and outputs rather than relying solely on perimeter or endpoint controls.

Practical implication: Security teams should treat training data, model files, and inference logs as governed assets that require validation and provenance checks.

Where agent governance becomes an identity problem

Once AI systems can autonomously call tools, retrieve data, or trigger workflows, they behave like governed software actors rather than passive features. That creates an identity boundary: the agent needs scoped authority, its actions need traceability, and its access needs to be limited to the minimum required for the task. If that governance is absent, the model layer becomes a route to unauthorised data access or unsafe system actions. This is where AI application security intersects with IAM, PAM, and NHI governance, because the question is no longer only what the model says but what the agent is allowed to do.

Practical implication: Define and enforce explicit authorisation boundaries for AI agents before they are allowed to access tools or sensitive systems.

Threat narrative

Attacker objective: The attacker aims to manipulate AI behaviour or abuse agent authority in order to extract data, distort outputs, or trigger unsafe downstream actions.

1. Entry occurs through adversarial prompts, poisoned data, or malicious model inputs that influence the AI system's behaviour during inference or fine-tuning.
2. Credential or authority abuse follows when the agent or model is allowed to call tools, access data, or execute workflows beyond the intended scope of the task.
3. Impact is achieved through data leakage, model manipulation, unsafe outputs, or misuse of connected systems that the AI is trusted to operate on.

Breaches seen in the wild

• Meta Muse agent hijack 2026: An undocumented Muse setting let local malware hijack Meta's personal AI agent, steal its authentication material and abuse user access.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI application security is now a governance problem, not just a testing problem. Traditional AppSec can still find familiar software defects, but it cannot by itself govern model behaviour, tool use, or delegated agent actions. The shift described in this article is structural: AI systems introduce a control plane that spans data, model, and runtime authority, which means security teams must manage what the system is allowed to do, not just what code it runs. The practitioner takeaway is that governance has to move upstream into design and authorisation.

Agent authority is the new boundary that conventional application security does not define. When an AI system can retrieve data, trigger actions, or chain tools together, its risk profile resembles a privileged software actor more than a static application feature. That is where identity governance becomes relevant, because the agent needs scoped permissions, lifecycle control, and traceability for every delegated action. The practitioner conclusion is that agent access should be designed and reviewed like a privileged identity, not treated as an incidental integration.

Model governance debt is becoming a real security liability. The article correctly shows that AI systems accumulate risk through data drift, model updates, and changing behaviour over time. That creates a governance debt when organisations deploy models faster than they can validate inputs, outputs, and downstream effects. The practical conclusion is that AI assurance must become continuous, or control gaps will accumulate faster than point-in-time reviews can resolve them.

Prompt injection and model theft are symptoms of a broader trust boundary failure. These threats matter because they exploit the assumption that model inputs, outputs, and connected tools are trustworthy once the system is deployed. In reality, AI systems operate across untrusted interaction surfaces, and that means trust must be continuously re-established at runtime. The practitioner conclusion is that AI security programmes need runtime controls that assume the model will be probed, manipulated, and chained to other systems.

From our research library:

• 67% of organisations still rely heavily on static credentials despite the risks they pose to agentic AI deployments, according to the 2026 Infrastructure Identity Survey.
• 53% of security leaders expect AI to run major portions of their infrastructure autonomously within the next three years, according to the 2026 Infrastructure Identity Survey.
• Read next: Agentic AI Security Guide

What this signals

AI security posture now depends on governing runtime behaviour, not just scanning software artefacts. For practitioners, the key change is that model updates, prompt paths, and tool permissions all become part of the control surface. A security programme that still treats AI as another web workload will miss the points where AI systems actually fail.

Agentic AI identity will become a practical control domain rather than a conceptual one. As soon as agents can act on behalf of users or services, identity, authorisation, and lifecycle management move from theory to operations. The programme question is no longer whether agents should be governed, but which permissions they should never hold in the first place.

Static credentials remain a weak default in agentic environments: 67% of organisations still rely heavily on them despite the risks they pose to agentic AI deployments, according to the 2026 Infrastructure Identity Survey. That matters because agents and models amplify the blast radius of long-lived access, especially when credentials are reused across automation, pipelines, and connected tools.

For practitioners

• Define explicit agent authorisation scopes Document which data sources, tools, and workflows each AI agent may access, and deny everything outside that task scope by default.
• Add model and dataset integrity checks Validate training inputs, fine-tuning data, and model artefacts before promotion so tampering and poisoning are detected before deployment.
• Instrument prompt and output monitoring Log high-risk prompts, unusual tool calls, and suspicious outputs so security teams can investigate manipulation attempts and unsafe behaviour.
• Treat AI agents as governed identities Assign ownership, review access regularly, and remove stale permissions for agents that no longer need access to systems or data.

Key takeaways

• AI application security is increasingly about controlling model behaviour, agent authority, and data pathways, not only finding code flaws.
• Traditional AppSec can still catch familiar vulnerabilities, but it leaves the AI-specific attack surface under-governed if model and prompt risk are excluded.
• The strongest control shift is toward continuous governance, where access, inputs, outputs, and tool use are monitored as part of the security boundary.

Key terms

• Agent Governance: Agent governance is the set of policies, controls, and evidence required to manage autonomous software as a non-human identity. It covers consent, tool access, lifecycle review, audit logging, and revocation so that an agent remains bounded as its workflows change.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Model Poisoning: Model poisoning is the deliberate corruption of training or ingestion data so an AI system learns or behaves incorrectly. The attack targets the trust chain before or during model training, which means the resulting failure can persist into production outputs, decisions, and automated actions.
• AI Application Security: AI application security is the discipline of protecting software that uses AI to make decisions, generate code, or interact with data and services. It goes beyond model safety to cover prompts, APIs, generated logic, runtime behaviour, and the permissions those components use in production.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and agentic AI identity for teams building governed access models. It helps security practitioners translate identity controls into operational decisions for modern AI and automation programmes.