Topic starter
01/08/2026 10:47 pm
TL;DR: Upgrading from GPT-4o to GPT-4.1 reduced prompt-injection resistance from 94% to 71% in a customer eval harness, while the post argues that model updates can change refusal behavior, instruction-following, and tool calling in ways that break existing agent controls, according to Promptfoo. Treating model upgrades as security changes is now mandatory because model-level safety is not a security boundary.
Editorial analysis by NHI Mgmt Group, based on content published by Promptfoo: “Your model upgrade just broke your agent's safety”.
Key questions
Q: What breaks when a model upgrade changes agent refusal behaviour?
A: What breaks first is the assumption that prior safety testing still predicts current behaviour.
Q: Why do retrieved documents increase the risk of prompt injection in agents?
A: Because retrieved documents can carry instructions that look like context to the model, not like attacker input.
Q: What do security teams get wrong about model safety in agentic systems?
A: They often treat model-level safety as the boundary, when the real boundary is the application layer that controls tools, data access, and execution.
Practitioner guidance
- Pin model IDs and safety settings Do not deploy a moving target such as latest.
- Re-run prompt-injection and tool-abuse tests on every upgrade Test direct injection, indirect injection through retrieved documents, multi-turn escalation, and suspicious tool-call behaviour before production rollout.
- Add application-layer guardrails around tools and RAG Validate output schemas, filter retrieved text, and enforce tool allowlists so the model cannot directly execute untrusted instructions from documents or user input.
Bottom line: A model upgrade can change the effective security posture of an agent even when the surrounding code stays the same.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
01/08/2026 10:56 pm
Model upgrades are security changes, not just product changes. The core mistake is assuming safety behaviour transfers across versions. Promptfoo’s example shows that refusal style, literalness, and tool planning can shift enough to change the attack surface even when the application code stays the same. Practitioners should manage model upgrades as controlled security events, with regression testing and sign-off before release.
A question worth separating out:
Q: How should organisations govern model upgrades for AI agents?
A: Organisations should treat each upgrade as a controlled security change with version pinning, regression tests, and explicit review of refusal style, tool calling, and retrieval handling. That approach turns behavioural drift into a managed change event instead of a silent control failure.
👉 Read our full editorial: Model upgrades can weaken agent security more than teams expect