TL;DR: Autonomous red teaming is being positioned as a way to stress-test LLMs and agentic systems before attackers do, while OWASP updates on excessive agency, system prompt leakage, and RAG weaknesses show why traditional red teaming leaves blind spots, according to Lasso Security and OWASP. The real issue is not just model testing, but the lack of clear ownership and benchmarks for systems that now change behaviour, expose prompts, and pull external data at runtime.
At a glance
What this is: NHI Mgmt Group's analysis, drawing on Lasso Security's blog post: Autonomous red teaming is filling a benchmark and ownership gap in LLM security as OWASP updates expose risks in excessive agency, prompt leakage, and retrieval weaknesses.
Why it matters: For IAM, NHI, and AI governance teams, the issue is that runtime AI behaviour changes the control problem from static testing to continuous identity and access oversight.
Context
LLM security is no longer just about model quality or content filtering. As generative AI systems take on more runtime decision-making and external data access, the control problem shifts toward who owns the risk, what gets tested, and whether the security programme can keep pace with changing behaviour.
That gap is especially visible where traditional red teaming was built for stable systems and human-paced remediation. In AI and LLM deployments, attack surfaces change with prompt logic, retrieval sources, and agentic features, which means governance now has to account for how systems act as much as what they produce.
Key questions
Q: What breaks when LLMs are red-teamed only with traditional manual methods?
A: Manual red teaming misses the combinations of prompts, retrieval paths, and tool interactions that emerge only at runtime. That creates blind spots in systems that can change behaviour quickly, especially when releases are frequent and external data sources are involved. Continuous autonomous testing is needed when the attack surface changes faster than review cycles.
Q: Why do excessive agency and prompt leakage matter for AI governance?
A: Excessive agency turns an LLM system into a governed actor that can affect downstream workflows, while prompt leakage exposes the instructions that shape those actions. Together they shift the control problem from model output quality to delegated authority, hidden logic, and behaviour at runtime. That is an identity governance issue as much as a security one.
Q: What are the signs that RAG is becoming a security problem?
A: Warning signs include weak control over retrieval sources, unclear provenance for indexed content, and responses that reflect data the system should not have reached. If attackers can influence source material or connectors, the retrieval layer can undermine the model even when the model itself is intact. Governance has to extend to the full data path.
Q: How should teams govern AI systems that can take actions as well as generate outputs?
A: Treat the agent as a governed actor, not just a model output stream. Require action-level logging, tool-call traceability, authorization boundaries, and approval gates before the system can write to records or invoke downstream tools. If an AI system can change state, its authority must be scoped, monitored, and revocable like any other privileged non-human identity.
Technical breakdown
Why autonomous red teaming changes the LLM testing model
Autonomous red teaming uses automated adversarial agents to probe an LLM or GenAI application continuously, rather than relying only on periodic human-led assessments. The value is not just speed. It is the ability to exercise large combinations of prompts, retrieval paths, and application states that manual teams rarely cover at scale. That matters because many LLM risks are interaction-dependent: they only appear when prompts, tools, data sources, and guardrails collide in a specific sequence. In that sense, autonomous testing is less a replacement for human red teaming than a coverage mechanism for systems whose behaviour changes too quickly for narrow test cycles.
Practical implication: Treat autonomous red teaming as coverage for dynamic AI behaviour, not as a substitute for governance ownership or remediation.
Excessive agency and prompt leakage create identity-adjacent risk
OWASP’s excessive agency category reflects a shift from passive model output risk to systems that can take meaningful actions. Once an AI system can decide or execute within a workflow, security teams must evaluate the permissions, guardrails, and escalation paths attached to that behaviour. System prompt leakage is a separate but related problem: exposed instructions can let attackers shape behaviour, bypass constraints, or infer sensitive operational logic. Both issues matter to identity governance because they show that control is no longer only about authentication. It is about what the system is allowed to do, reveal, and infer at runtime.
Practical implication: Review AI permissions, hidden instructions, and execution boundaries together rather than as separate controls.
RAG weaknesses widen the attack surface beyond the model
Retrieval-augmented generation depends on external content sources, which means the security boundary extends beyond the model itself. If retrieval sources are weakly governed, attackers can poison inputs, influence outputs, or expose sensitive material through indirect responses. That makes the integrity of data sources, indexing paths, and access controls part of the AI security model. For practitioners, the critical point is that a vulnerable retrieval layer can undermine an otherwise well-tuned model because the system is only as trustworthy as the data it can fetch at runtime.
Practical implication: Govern retrieval sources, access scope, and data integrity as part of the AI security perimeter.
Threat narrative
Attacker objective: The attacker wants to manipulate LLM behaviour, expose hidden instructions or data, and trigger unsafe or untrusted outputs that reduce system assurance.
- Entry occurs through prompts, retrieval sources, or agent interactions that are allowed to influence LLM behaviour at runtime. Credentialed access is not required when the attacker can shape the system through trusted interfaces.
- Escalation happens when excessive agency, leaked system prompts, or weak retrieval controls let the attacker influence decisions, expose hidden logic, or steer downstream actions beyond intended scope.
- Impact follows when the model returns sensitive information, bypasses guardrails, or behaves in ways that undermine business workflows and security controls.
Breaches seen in the wild
- DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.
- Langflow Flodrix botnet 2025: Attackers used Langflow's unauthenticated code endpoint to dump AI servers' environment variables and enlist them in the Flodrix botnet.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Autonomous red teaming fills a coverage gap, not a governance gap: Traditional red teaming was designed for bounded systems with stable interfaces and slower change cycles. LLM deployments evolve through prompts, retrieval sources, and application logic that shift faster than manual test programmes can follow. The implication is that benchmarking alone is insufficient unless ownership and remediation are also defined.
Excessive agency is the clearest sign that AI security has crossed into identity governance: Once an AI system can choose actions, invoke tools, or affect downstream systems, the question is no longer only model safety. It becomes a control problem around permitted behaviour, delegated authority, and runtime boundaries. Practitioners need to treat these systems as governed actors, not just software features.
System prompt leakage is a control-plane problem, not just a confidentiality issue: Exposed prompts can reveal operational logic, policy intent, and manipulation paths that attackers can reuse. That means prompt protection belongs alongside authorisation and workflow design, because leaked instructions can become a blueprint for control bypass. Security teams should treat prompt content as part of the system's sensitive operating model.
RAG creates identity and data trust coupling that most security programmes still separate: Retrieval makes external content part of the model's decision path, so weak source governance becomes a model integrity issue. This is where access control, source validation, and data provenance converge. The practitioner conclusion is simple: if the retrieval layer is not governed, the model cannot be trusted.
From our research library:
- AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
- Only 23% of IT leaders were very confident in their organisation's ability to manage security and governance for GenAI deployments, according to a 2025 Gartner survey of 360 IT leaders.
- Read next: Agentic AI Security Guide
What this signals
Autonomous red teaming should be treated as a control-completeness check for AI programmes, not a one-time test event. LLM and GenAI systems can change through prompts, tools, and retrieval inputs faster than periodic reviews can keep up, so security teams need continuous validation rather than annual assurance windows.
AI security now depends on the trust boundary around the model, not just the model itself. If retrieval sources, prompts, and action permissions are not governed together, the system can be secure in isolation and still unsafe in production.
AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026. That pattern reinforces why identity, secrets, and runtime AI governance must be managed as one problem, not separate ones.
For practitioners
- Define an owner for AI security benchmarking Assign a named security owner for each LLM or GenAI deployment so red-team findings do not stall between model, platform, and application teams.
- Test excessive agency as a permissions problem Map which AI actions are truly allowed at runtime, then verify those permissions against the workflows the system can initiate without human approval.
- Inventory system prompts and guardrail logic Treat prompts, policy instructions, and hidden control logic as governed assets that must be reviewed, protected, and version-controlled.
- Validate retrieval source integrity Check which external sources, indexes, and connectors feed LLM responses, then restrict and monitor them with the same discipline used for sensitive data access.
- Use autonomous red teaming on pre-production and live systems Run continuous attack simulations against both staged and deployed AI applications so changes in prompts, tools, or retrieval paths are tested before attackers exploit them.
Key takeaways
- Traditional red teaming leaves gaps when LLM behaviour shifts through prompts, retrieval, and tool use at runtime.
- The article’s core evidence is that OWASP risk areas such as excessive agency, system prompt leakage, and RAG weaknesses now need continuous testing.
- Security teams need ownership, permission mapping, and retrieval governance before AI systems are trusted in production.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST AI RMF sets the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | The article centers on AI systems taking action and crossing governance boundaries. |
| ASI06 — Memory & Context Poisoning | Prompt leakage and retrieval manipulation both distort agent context and behaviour. | |
| Recommendation — Map runtime AI permissions to ASI03 and restrict actions the system can initiate without approval. Test memory and context paths for poisoning and remove untrusted inputs from agent decision flows. | ||
| OWASP Non-Human Identity Top 10 | NHI-04 — Insecure Authentication | LLM and AI platform access depends on identities, tokens, and controlled runtime access. |
| NHI-02 — Secret Leakage | The article explicitly references leaked prompts, API keys, and hidden logic exposure patterns. | |
| Recommendation — Audit AI platform authentication paths and ensure only approved identities can reach model and retrieval services. Scan AI systems for exposed secrets and treat prompts, keys, and connectors as sensitive assets. | ||
| NIST AI RMF | GOVERN — AI Governance and Accountability | Ownership and governance gaps are a core theme of the article. |
| Recommendation — Assign clear AI governance ownership for testing, findings, and remediation across the deployment lifecycle. | ||
Key terms
- Automated red-teaming: Automated red-teaming is the use of adversarial test generation to find how an AI model or agent fails under pressure. It goes beyond manual review by systematically probing prompt injection, goal drift, unsafe outputs, and other repeatable behavioural weaknesses before production use.
- Excessive agency: A condition where an AI system is given more operational authority than its task requires. The risk is not just poor output. It is that mistakes, manipulation, or compromise can produce destructive actions at machine speed across the systems the agent can reach.
- System Prompt Leakage: System prompt leakage is the exposure of hidden prompt content to users or attackers. The real security problem is usually not the prompt itself, but the secrets, policy logic, and internal architecture details placed inside it. If those details are sensitive, they should live in code or secrets management instead.
- Retrieval-augmented Generation: Retrieval-augmented generation is a pattern where an AI model pulls external information before generating output. The security challenge is that access rules can weaken when data is chunked, embedded, cached, or reused, so source permissions may not automatically follow the content into the model's context.
Deepen your knowledge
NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.
Published by the NHIMG editorial team on June 9, 2026.
Updated on October 10, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org