GenAI model security needs continuous controls, not scan-and-ship

At a glance

What this is: This is a technical blueprint for GenAI model security that argues the real control gap is continuous protection across prompts, data pipelines and agentic actions.

Why it matters: It matters because IAM, PAM and security teams now have to govern AI systems that can read data, call tools and make decisions without fitting legacy application control assumptions.

By the numbers:

• Only 24% of current GenAI projects incorporate security considerations.
• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.

👉 Read Akto's technical blueprint for continuous GenAI model protection

Context

GenAI security is not just an application security problem. Once models can read documents, retrieve context, call APIs and act on behalf of users, the control plane shifts from code integrity to runtime governance, data handling and authorization boundaries. That makes primary keyword coverage around GenAI model security relevant to IAM, NHI and agentic AI programmes alike.

The article frames a familiar enterprise pattern: adoption outruns governance, then teams discover that static scans and perimeter tools cannot express trust in probabilistic systems. The identity bridge is real here because AI agents, MCP tools and model-connected services behave like non-human identities with access, lifecycle and audit requirements that legacy controls often were never designed to manage.

Key questions

Q: What breaks when GenAI systems are secured like ordinary applications?

A: They miss the main risk surface. GenAI systems can be manipulated through language, retrieved content and tool outputs, so a clean code scan does not prove safe behaviour. If teams rely only on static controls, they will miss prompt injection, model drift and unauthorised actions that appear only at runtime.

Q: Why do AI agents complicate existing IAM and PAM controls?

A: AI agents complicate IAM and PAM because they often inherit delegated credentials, operate across multiple systems, and keep acting after the initial approval moment has passed. Human session assumptions, periodic reviews, and static privilege models do not reflect that behaviour. The result is a governance gap between what was granted and what the agent can actually do.

Q: How do security teams know runtime AI guardrails are actually working?

A: Look for blocked poisoned inputs, flagged anomalous outputs, and traceable enforcement before responses reach users or downstream systems. If controls only inspect prompts or only inspect outputs, they leave a gap that attackers can exploit through manipulated data sources or tool responses.

Q: Which frameworks help teams operationalise AI risk governance?

A: The NIST AI Risk Management Framework is the clearest reference point because it emphasises govern, map, measure, and manage as ongoing functions. Teams should use it to connect policy, evidence, and monitoring rather than treating AI governance as a one-time compliance checkpoint.

Technical breakdown

Prompt injection and output manipulation in GenAI systems

Prompt injection is the abuse of natural language context to redirect a model away from intended behaviour. The input may come from a user, a document, a webpage or a tool response, which means the attacker does not need code execution to influence the system. Output manipulation is the related problem where the model is induced to leak sensitive information, generate unsafe instructions or trigger harmful downstream actions. The key architectural issue is that the model treats context as data and instruction at the same time, so trust boundaries collapse unless inputs, outputs and tool responses are independently filtered and validated.

Practical implication: treat every external prompt source as untrusted and enforce input sanitisation, output validation and tool-response filtering at runtime.

Model poisoning and AI supply chain risk

Model poisoning targets the training and fine-tuning pipeline rather than inference. If an attacker can alter datasets, embeddings, checkpoints or code that feeds the model, they can introduce hidden behaviours that only activate under specific triggers. This is hard to catch because the model may appear normal during testing and fail only in production. The supply chain dimension matters because many GenAI stacks rely on open-source datasets, third-party embeddings and reused model assets. That makes provenance, version control and pipeline access control central governance requirements, not optional hygiene.

Practical implication: restrict who can modify training assets, verify provenance for datasets and checkpoints, and test for behavioural drift before release.

Agentic workflow abuse and AI-to-tool authorisation

Agentic AI becomes materially different from a passive LLM once it can call APIs, run code or trigger business actions. At that point the security question is no longer only what the model says, but what the system does next. Risks emerge when tool permissions are inherited from human workflows, when sensitive actions are not gated, or when one agent can impersonate another inside a multi-agent chain. This is where identity governance, authorisation and audit logging intersect directly with AI security. The model itself may be bounded, but the connected action layer can still create real-world impact.

Practical implication: assign per-agent permissions, require human approval for irreversible actions, and log every tool call for audit and containment.

Threat narrative

Attacker objective: The attacker wants to turn a trusted GenAI system into a channel for data exposure, policy bypass or unauthorised action execution.

1. Entry occurs through malicious prompts, poisoned training inputs or compromised tool responses that the model accepts as context.
2. Escalation follows when the model is induced to reveal sensitive data, bypass intended guardrails or invoke connected tools with excessive trust.
3. Impact lands when the agent sends emails, changes records, leaks credentials or triggers business processes the organisation never intended to automate unsupervised.

NHI Mgmt Group analysis

GenAI model security is becoming a control-plane problem, not an AppSec add-on. The article is right to separate probabilistic model behaviour from deterministic application risk. Once the model can read, reason and act, the relevant questions become who can influence its context, what it can touch, and which actions need approval. That reframes security architecture around runtime control, auditability and least privilege, which is where IAM and PAM disciplines become directly relevant to AI operations.

Shadow AI creates the same governance debt that shadow IT created, but with faster consequence. Unapproved models, agents and MCP tools can appear inside products, user workflows and vendor integrations without being catalogued. That means the organisation may already have an AI control problem before it has an AI programme. The specific failure mode is not simply lack of visibility, but lack of inventory coupled with no enforceable policy boundary. Practitioners should treat undiscovered GenAI assets as unmanaged identities with unknown access scope.

Continuous runtime protection is the right operating assumption for GenAI because static review cannot measure live behaviour. Traditional scan-and-ship logic assumes the risk is stable at release time. GenAI systems can change outcomes as prompts, retrieval sources and tools change, so the security target moves every time the environment changes. The named concept here is behavioural drift exposure: the gap between what a model was tested to do and what it actually does in production. Security teams need controls that observe and constrain live behaviour.

AI agents should be governed as non-human identities when they can invoke tools or perform business actions. That intersection matters because the access decision is no longer about a person authenticating into a system. It is about whether a software entity has the right to retrieve data, execute actions and preserve evidence. Existing identity governance models struggle when agent permissions are inherited informally or when audit trails do not separate model reasoning from tool execution. The practitioner conclusion is clear: if an AI system can act, it needs an identity boundary.

What this signals

Behavioural drift exposure will become a standard GenAI governance term because production risk changes whenever prompts, retrieval sources or tools change. Teams that still think in quarterly review cycles will miss the real operating rhythm of AI systems, which is continuous and highly dependent on runtime context. The right reference points are NIST AI RMF and the OWASP Agentic AI Top 10, not legacy application hardening alone.

AI programmes now need identity-aware guardrails for software entities that can act on behalf of users. Once a model can invoke APIs or complete tasks, permissioning and evidence retention matter as much as prompt quality. That makes agent identity, tool-level logging and approval design part of the security architecture, not an afterthought.

The practical shift is toward continuous verification of what models and agents can see, do and disclose. Organisations that build discovery, testing and enforcement into the same control loop will be better positioned to contain shadow AI and limit blast radius when a model or agent behaves unexpectedly.

For practitioners

• Implement continuous GenAI asset discovery Catalogue every LLM, AI agent and MCP tool in use, including shadow AI surfaced through user workflows and third-party integrations. Tie discovery to ownership so each asset has an accountable service or team. The goal is real-time inventory, not a spreadsheet that ages out in days.
• Enforce runtime input and output controls Inspect prompts, retrieved content and tool responses before they reach the model or downstream systems. Block embedded instructions, restrict system prompt exposure and validate outputs before they trigger any action or are shown to users.
• Separate model access from action authority Give each agent only the permissions needed for its task, and gate irreversible actions such as payments, record changes or external communications behind human approval. Use tool-level logging so every action can be reconstructed during investigation.
• Control training pipeline integrity Restrict who can modify datasets, embeddings and fine-tuning inputs, then version and verify those assets before release. Add behavioural testing for trigger-based failures so poisoned data is detected before production exposure.
• Run continuous adversarial testing Red team production-like prompts, documents and agent workflows on an ongoing basis, not only before launch. Re-test after model updates, connector changes or retrieval changes because each of those can change the effective attack surface.

Key takeaways

• GenAI security fails when teams treat models like deterministic applications instead of runtime systems with shifting behaviour.
• The evidence shows adoption is outpacing governance, with AI agents already acting beyond intended scope in most organisations.
• The control answer is continuous discovery, runtime guardrails, and identity-aware authorisation for every model and agent that can act.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Model Poisoning: Model poisoning is the deliberate corruption of training or ingestion data so an AI system learns or behaves incorrectly. The attack targets the trust chain before or during model training, which means the resulting failure can persist into production outputs, decisions, and automated actions.
• Agentic workflow: An agentic workflow is a sequence of tasks executed by an AI agent with some level of tool access and decision authority. In security terms, the workflow matters because it can span multiple systems, identities, and permissions, which makes attribution and revocation harder than with ordinary automation.
• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.

What's in the full article

Akto's full blog covers the operational detail this post intentionally leaves for the source:

• Practical examples of input, output and retrieval-layer sanitisation that teams can adapt for production GenAI systems.
• Step-by-step guidance on policy enforcement inside deployment pipelines and access systems for AI assets.
• Runtime guardrail patterns for monitoring model behaviour, blocking unsafe actions and logging tool calls.
• Implementation detail on discovery, inventory and posture management for LLMs, agents and MCP tools.

👉 Akto's full post covers the attack patterns, runtime controls and governance sequence in more implementation detail.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security and secrets management. It helps security and identity practitioners translate governance into controls that work across modern identity programmes.