Topic starter
05/07/2026 6:47 pm
TL;DR: AI systems shift behavior with context, prompts, model updates, and tool use, so static tests miss the failures that emerge only during interaction, according to Lakera. Security now has to govern behavior across the full AI application lifecycle, not just model outputs.
Editorial analysis by NHI Mgmt Group, based on content published by Lakera: “AI Red Teaming: The Art of Stress-Testing Non-Deterministic Systems”.
Key questions
Q: What breaks when AI red teaming is treated like traditional penetration testing?
A: Teams miss the AI-specific attack surface, especially prompt handling, output manipulation, retrieval poisoning, and tool misuse.
Q: Why do AI prompts create identity and data-security risk?
A: AI prompts create risk because they can carry sensitive content outside the original system’s protection boundary.
Q: How do security teams know if AI red teaming is working?
A: AI red teaming is working when testing finds real prompt injection paths, over-scoped integrations, and policy gaps before attackers do, and when fixes are re-tested successfully after model or workflow changes.
Practitioner guidance
- Map prompt-to-action pathways Identify where prompts influence tool selection, data access, or execution so red teaming can test the full decision chain, not only model output quality.
- Re-test after every behavioural change Schedule regression testing whenever the foundation model, system prompt, tools, or permissions change, because each change can alter the system’s security boundary.
- Scope red teaming to application context Test the specific AI workflow, business data, and allowed actions that matter in production instead of relying on generic jailbreak exercises.
Bottom line: AI red teaming is moving from point-in-time evaluation to continuous assurance because non-deterministic systems change with context, tools, and updates.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
05/07/2026 7:04 pm
Deterministic security assumptions collapse when the system can change behaviour at runtime. Traditional controls assume that the same input produces the same outcome, which is a poor fit for AI applications whose responses vary with phrasing, context, and model updates. That assumption fails because the application is no longer stable enough for point-in-time assurance to be meaningful. The implication is that identity and access programmes must stop treating AI behaviour as predictable by design and start treating it as a moving control surface.
A question worth separating out:
Q: What should organisations do when AI systems can send emails or write to databases?
A: Treat those capabilities as delegated actions with explicit approval boundaries, logging, and review. Once an AI system can act in the environment, red teaming must test how it can be steered into using those permissions in unsafe ways. The question is not just what it generates, but what it can cause to happen.
👉 Read our full editorial: AI red teaming shows why deterministic security no longer fits