Notifications
Clear all
Topic starter
09/07/2026 12:35 pm
TL;DR: AI red teaming simulates adversarial prompts, vector-store poisoning, and function misuse to expose how large language models can leak sensitive data or violate policy, with Cornell research and related studies showing that small changes in context can trigger measurable failures. The governance gap is no longer hypothetical: inferential leakage and oversharing require continuous testing, not one-time validation.
NHIMG editorial — based on content published by Knostic: Fast Facts on Red Teaming AI
By the numbers:
- A 2025 study found indirect prompt injection succeeded in 92% of assistant-role contexts and 86% of system-role contexts, compared with 52% in standard user-role prompts.
- A 2024 Cornell University study found that just five poisoned documents could steer a retrieval-augmented model to attacker-chosen responses 90% of the time.
- A 2025 report found attack success rates of 71% and 70% against AgentDojo and VWA-adv, showing how often fuzzy prompt attacks can defeat LLM-agent defenses.
Questions worth separating out
Q: How should security teams implement AI red teaming in enterprise environments?
A: Start by scoping the prompts, data sources, retrieval systems, and tool permissions the model can reach.
Q: Why do AI agents complicate IAM and NHI governance?
A: AI agents can read data, choose actions, and invoke tools, so their effective privileges are distributed across identities, connectors, and content.
Q: What breaks when retrieval-augmented generation is not governed tightly enough?
A: When retrieval scope is too broad, the model can surface sensitive material that users should not have seen, even if no one explicitly requested a protected file.
Practitioner guidance
- Scope the full AI attack surface Inventory prompts, plugins, retrieval connectors, shared documents, and tool APIs before you test.
- Test for inference-based leakage Red-team models for information that can be inferred, assembled, or resurfaced from benign-looking context, not just direct file access.
- Constrain model function calls Restrict which internal APIs, workflows, and permissions an agent can invoke, and verify those controls with adversarial prompts.
What's in the full article
Knostic's full analysis covers the operational detail this post intentionally leaves for the source:
- Step-by-step red-team workflow for scoping prompts, retrieval connectors, and tool APIs in enterprise AI.
- Detailed examples of prompt, retrieval, and function misuse tests that uncover specific leakage paths.
- Operational remediation guidance for tightening permissions, sensitivity labels, and retrieval controls.
- How the vendor maps red-team findings into governance workflows for Copilot and similar AI tools.
👉 Read Knostic's analysis of AI red teaming, LLM leakage, and governance →
AI red teaming and LLM leakage: where do controls fail?
Explore further
09/07/2026 12:41 pm
AI red teaming is becoming the practical control plane for LLM governance. Static review does not tell you how a model behaves when prompts, retrieval, and tool permissions collide under adversarial pressure. The article’s core evidence shows that leakage and misuse emerge from interactions, not just from code defects. Practitioners should treat red teaming as a standing governance function, not a one-off security exercise.
A few things that frame the scale:
- While 71% of IT teams have been advised on AI agent data access, only 47% of compliance teams, 39% of legal teams, and 34% of executives have the same visibility, according to AI Agents: The New Attack Surface report.
- Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation, according to the same report.
A question worth separating out:
Q: Who is accountable when an AI system leaks data through a prompt or tool call?
A: Accountability should sit with the team that owns the model, the retrieval layer, and the connected data sources, because the leak usually spans all three. Governance frameworks should require audit logs, access reviews, and change tracking so security, privacy, and application owners can trace how the exposure happened and prove remediation.
👉 Read our full editorial: AI red teaming exposes where LLMs leak data and break policy
