Model upgrades can weaken agent security more than teams expect

At a glance

What this is: Promptfoo shows that a model upgrade can materially change agent security behaviour, with prompt-injection resistance dropping from 94% to 71% in one customer eval.

Why it matters: IAM, NHI, and AI security teams should treat model version changes as control changes, because tool gating, refusal behaviour, and RAG safety can shift even when the application code does not.

Context

A model upgrade is not just a quality or cost decision. In agentic systems, the model sits inside the control path for instruction-following, refusal behaviour, and tool use, so a change in model family can alter the security posture of the whole workflow.

The risk is especially sharp where agents depend on retrieved content, long-running tasks, and connected tools. Those environments create an identity and privilege problem as well as an AI one, because the agent can act with delegated access while trusting content that should be treated as untrusted input.

Key questions

Q: What breaks when a model upgrade changes agent refusal behaviour?

A: What breaks first is the assumption that prior safety testing still predicts current behaviour. If a newer model becomes more literal or more willing to follow embedded instructions, indirect prompt injection and tool misuse can rise even when the application code has not changed. Teams need to revalidate the agent as a whole, not just the model benchmark.

Q: Why do retrieved documents increase the risk of prompt injection in agents?

A: Because retrieved documents can carry instructions that look like context to the model, not like attacker input. Once the model treats those instructions as authoritative, the agent can be steered into unsafe outputs or tool calls. Instruction and data separation is essential when RAG is connected to actions or downstream systems.

Q: What do security teams get wrong about model safety in agentic systems?

A: They often treat model-level safety as the boundary, when the real boundary is the application layer that controls tools, data access, and execution. A model can refuse harmful text and still trigger unsafe behaviour through a tool call. Security has to be enforced before execution, not only during generation.

Q: How should organisations govern model upgrades for AI agents?

A: Organisations should treat each upgrade as a controlled security change with version pinning, regression tests, and explicit review of refusal style, tool calling, and retrieval handling. That approach turns behavioural drift into a managed change event instead of a silent control failure.

Technical breakdown

Why model upgrades change prompt-injection resistance

Model families are trained with different trade-offs between helpfulness, refusal style, literal instruction-following, and tool planning. A newer model may appear safer in benchmark terms while becoming more willing to follow embedded instructions inside retrieved text, which is exactly why indirect prompt injection remains hard to eliminate. The underlying issue is that model-level safety is behavioural, not a hard security boundary. Once the model is embedded in an agent that reads documents and can call tools, those behavioural shifts change how the whole application handles untrusted input.

Practical implication: Pin model versions and re-run adversarial tests whenever you change the model, not only when you change prompts or tools.

How tool calling turns model behaviour into a security issue

Tool calling creates a gap between text safety and execution safety. A model can refuse harmful text output and still produce a dangerous tool call if the surrounding system accepts the action. That is why agent security depends on application-layer controls such as allowlists, parameter validation, scoped credentials, and execution-time checks. In practice, the model proposes actions, but the system must decide whether those actions are permitted. Without that separation, prompt injection can move from a content problem to an access problem.

Practical implication: Gate every tool call outside the model and require explicit authorization checks before execution.

Why RAG and retrieved documents amplify risk

Retrieval-Augmented Generation expands the attack surface because retrieved documents can carry malicious instructions, hidden commands, or manipulation designed to override system intent. If the agent treats retrieved text as trusted context, the attacker gains a path from content injection to tool abuse. This is not a niche edge case. Any workflow that mixes user prompts, retrieved documents, and action-capable tools needs instruction and data separation, output validation, and scanning before execution. The governing principle is simple: retrieved content is evidence, not authority.

Practical implication: Filter retrieved text for instructions and validate outputs before any downstream tool or workflow action.

Threat narrative

Attacker objective: The attacker wants to steer an otherwise legitimate agent into unsafe execution, data exposure, or unauthorized tool use.

1. Entry occurs when an attacker places malicious instructions into user input, retrieved documents, or tool output that the agent later consumes.
2. Credential or authority abuse follows when the model accepts the injected instruction and reaches a tool-bearing path with delegated access.
3. Impact occurs when the agent executes an unauthorized action, leaks data, or performs an unsafe tool call that the security team did not intend.

Breaches seen in the wild

• EchoLeak (Microsoft 365 Copilot) 2025: A crafted email could make Microsoft 365 Copilot leak data from its context with no click, a zero-click prompt injection fixed as CVE-2025-32711.
• ForcedLeak (Salesforce Agentforce) 2025: A critical Agentforce flaw let attackers steer an AI agent through a Web-to-Lead form to leak CRM data via an expired allowlisted domain.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Model upgrades are security changes, not just product changes. The core mistake is assuming safety behaviour transfers across versions. Promptfoo’s example shows that refusal style, literalness, and tool planning can shift enough to change the attack surface even when the application code stays the same. Practitioners should manage model upgrades as controlled security events, with regression testing and sign-off before release.

Model safety is not the same as agent security. A model can still be text-safe while remaining operationally unsafe once it can read retrieved content or call tools. That distinction matters for IAM and NHI teams because the real risk is not what the model says, but what the agent is allowed to do with delegated access. The boundary has to move from model behaviour to application enforcement.

Instruction and data separation is the governing concept this article sharpens. Retrieved text should never be allowed to compete with system instructions on equal footing. In agentic workflows, that separation is the difference between a model that can summarise untrusted content and a system that can be socially engineered into acting on it. Treating retrieval as authority creates a verification trust gap that adversaries can exploit.

Agent privilege must be treated as ephemeral and revocable. If a model upgrade changes tool use, the surrounding control plane has to absorb that change without expanding standing access. That means the operational question is not whether the model is aligned in the abstract, but whether the agent can be stopped, scoped, and observed at execution time. Security teams should expect model drift and govern around it.

The named concept here is model security drift. That is the gap between a model’s previous and current behaviour after an upgrade, where inherited controls no longer match the new refusal and instruction-following profile. The concept matters because many agent controls are tuned to a specific model behaviour, so an upgrade can silently invalidate the assumptions those controls depend on. Practitioners need version-aware governance, not static approval.

What this signals

Model upgrades create a versioned governance problem for AI agents. If refusal style, literalness, or tool planning changes after a release, the surrounding control stack must be revalidated as part of the deployment process.

Model security drift: the behavioural gap between one model version and the next can invalidate agent controls that were tuned to earlier refusal and instruction-following patterns. For practitioners, that means model governance, tool gating, and retrieval filtering have to be tied to release management, not left as static safeguards.

For practitioners

• Pin model IDs and safety settings Do not deploy a moving target such as latest. Lock the model version, record safety thresholds in code, and require approval for any model family or setting change.
• Re-run prompt-injection and tool-abuse tests on every upgrade Test direct injection, indirect injection through retrieved documents, multi-turn escalation, and suspicious tool-call behaviour before production rollout.
• Add application-layer guardrails around tools and RAG Validate output schemas, filter retrieved text, and enforce tool allowlists so the model cannot directly execute untrusted instructions from documents or user input.
• Log and alert on injection signals and suspicious tool attempts Capture session context, retrieved document IDs, tool names, and denial events so repeated injection patterns can trigger containment and regression updates.
• Treat agent credentials as scoped execution privileges Use least-privilege credentials per tool and route, then revoke or rotate them if an injection event suggests the agent attempted unsafe actions.

Key takeaways

• A model upgrade can change the effective security posture of an agent even when the surrounding code stays the same.
• Prompt injection becomes more dangerous when retrieved content and tool calls share the same execution path.
• Version pinning, regression testing, and execution-time tool controls are the practical response to behavioural drift.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Model Security Drift: Model security drift is the change in an AI model’s safety and instruction-following behaviour after an upgrade or configuration change. The drift matters because controls tuned to one version may no longer match the next version’s refusal style, literalness, or tool-planning behaviour.
• Tool Gating: A control that restricts which external actions or connectors an AI system may invoke and under what conditions. It reduces the chance that malicious prompts or contaminated context can turn a model into an unsafe executor of privileged operations.
• Retrieval-augmented Generation: Retrieval-augmented generation is a pattern where an AI model pulls external information before generating output. The security challenge is that access rules can weaken when data is chunked, embedded, cached, or reused, so source permissions may not automatically follow the content into the model's context.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and machine identity security. It helps practitioners connect delegated access, lifecycle control, and security review across identity-led programmes.