Notifications
Clear all
Topic starter
02/08/2026 12:31 pm
TL;DR: Teams that ship LLM features often blur AI safety and AI security, but the article shows why that creates exploitable gaps, from autonomous agents deleting production data to prompt injection and hidden instructions, according to Promptfoo and cited incident reports. The practical lesson is that helpful outputs and secure runtime behaviour need separate testing, controls, and accountability.
NHIMG editorial — based on content published by Promptfoo: AI safety and AI security are different problems with different controls
By the numbers:
- IBM's 2025 report puts the global average cost of a data breach at $4.44M.
Questions worth separating out
Q: How should security teams test AI systems for safety and security separately?
A: Run two evaluation tracks.
Q: Why do AI agents complicate existing IAM and PAM controls?
A: AI agents complicate IAM and PAM because they often inherit delegated credentials, operate across multiple systems, and keep acting after the initial approval moment has passed.
Q: What breaks when prompt injection is not governed like an access problem?
A: The organisation may treat malicious text as a harmless message, even though it can steer an agent into exposing data or taking privileged actions.
Practitioner guidance
- Separate safety testing from security testing Build distinct evaluation tracks for harmful outputs and adversarial manipulation.
- Treat AI agents as privileged non-human identities Assign scoped, short-lived credentials to every agent that can call tools or access data.
- Sandbox external content before the model can act on it Strip executable instructions from emails, calendar items, README files, and retrieved pages before they reach the model context.
What's in the full article
Promptfoo's full analysis covers the operational detail this post intentionally leaves for the source:
- Concrete incident references and evaluation examples for Replit, Grok, Gemini, Cursor, and Amazon Q
- Practical test design for separating safety red teaming from security red teaming
- Implementation guidance for prompt sanitisation, tool containment, and immutable logging
- A detailed breakdown of OWASP LLM Top 10 mappings and related control choices
👉 Read Promptfoo's analysis of AI safety versus AI security in LLM systems →
AI safety and AI security: are your controls keeping up?
Explore further
02/08/2026 1:28 pm
AI safety and AI security are separate governance disciplines, not two labels for the same risk. Safety failures create harmful outputs, but security failures create unauthorised action paths through tools, data, and credentials. Organisations that collapse the distinction tend to overinvest in content filters while underinvesting in access control and runtime containment. The right governance model is to treat model behaviour and system privilege as different control surfaces.
A question worth separating out:
Q: Which controls matter most when an LLM can access production systems?
A: Prioritise least privilege, short-lived credentials, tool-level allowlists, immutable logging, and human approval for irreversible actions. Those controls matter more than output filtering alone because the main risk is not just what the model says. It is what the model can be induced to do through authorised integrations.
👉 Read our full editorial: AI safety and AI security are different problems with different controls
