Topic starter
01/08/2026 10:47 pm
TL;DR: Prompt-injection resistance fell from 94% to 71% when a customer agent moved from GPT-4o to GPT-4.1, according to Promptfoo. Agent controls need fresh regression testing because refusal, tool calling, and retrieval handling can drift after a model swap.
Editorial analysis by NHI Mgmt Group, based on content published by Promptfoo: “Your model upgrade just broke your agent's safety”.
Key questions
Q: Why do retrieved documents increase the risk of prompt injection in agents?
A: Because retrieved documents can carry instructions that look like context to the model, not like attacker input.
Q: What do security teams get wrong about model safety in agentic systems?
A: They often treat model-level safety as the boundary, when the real boundary is the application layer that controls tools, data access, and execution.
Q: What breaks when a model upgrade changes agent refusal behaviour?
A: What breaks first is the assumption that prior safety testing still predicts current behaviour.
Practitioner guidance
- Pin model IDs and safety settings Do not deploy a moving target such as latest.
- Re-run prompt-injection and tool-abuse tests on every upgrade Test direct injection, indirect injection through retrieved documents, multi-turn escalation, and suspicious tool-call behaviour before production rollout.
- Add application-layer guardrails around tools and RAG Validate output schemas, filter retrieved text, and enforce tool allowlists so the model cannot directly execute untrusted instructions from documents or user input.
Bottom line: A model upgrade can change the effective security posture of an agent even when the surrounding code stays the same.
Glossary terms: Prompt Injection (Agentic), Model Security Drift, Tool Gating, Retrieval-augmented Generation
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
This topic was modified 7 hours ago by NHI Mgmt Group
01/08/2026 10:56 pm
Model upgrades are security changes, not just product changes. The core mistake is assuming safety behaviour transfers across versions. Promptfoo’s example shows that refusal style, literalness, and tool planning can shift enough to change the attack surface even when the application code stays the same. Practitioners should manage model upgrades as controlled security events, with regression testing and sign-off before release.
A question worth separating out:
Q: How should organisations govern model upgrades for AI agents?
A: Organisations should treat each upgrade as a controlled security change with version pinning, regression tests, and explicit review of refusal style, tool calling, and retrieval handling. That approach turns behavioural drift into a managed change event instead of a silent control failure.
👉 Read our full editorial: Model upgrades can weaken agent security more than teams expect