AI red teaming is now a governance issue for LLM applications

At a glance

What this is: This is an analysis of how AI red teaming tests deployed LLM applications against realistic attack techniques, with the key finding that the real risk sits in connected tools, retrieval pipelines, and agentic workflows rather than model output alone.

Why it matters: It matters because IAM, PAM, and NHI teams increasingly need to govern the permissions, tool access, and runtime boundaries around AI applications, not just the model layer.

By the numbers:

• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes.

👉 Read Novee’s analysis of how modern security teams use AI red teaming

Context

AI red teaming is the discipline of testing a deployed AI application the way an attacker would, using prompt injection, jailbreaks, tool abuse, and indirect instruction attacks. The important shift is that the application, its integrations, and its business logic are the attack surface, which means security teams have to evaluate runtime behavior rather than model quality alone.

That distinction matters for IAM, PAM, and NHI governance because LLM-powered applications increasingly act through API keys, service accounts, retrieved content, and delegated tools. Once a model can read data or call systems, access control, privilege boundaries, and identity lifecycle controls become part of AI security whether teams planned for that or not.

Novee’s framing is typical of the current market: the strongest AI testing programmes now treat language-based manipulation as a live control problem, not a one-time model assessment.

Key questions

Q: What breaks when AI security stops at model scanning?

A: Model scanning helps identify tampering and unsafe dependencies before deployment, but it does not address runtime misuse. Once the system is live, prompt injection, unsafe tool use, and manipulated responses can still drive harmful behaviour. Without runtime controls, the most important security decisions happen after the pre-check has already passed.

Q: When does AI red teaming need to move from periodic testing to continuous testing?

A: Continuous testing becomes necessary when model updates, new integrations, or prompt changes can alter agent behaviour without a code rewrite. If a new API, MCP server, or data source expands the reachable surface, a calendar-based test is no longer enough. Change-triggered red teaming keeps the control posture aligned with the system’s actual identity scope.

Q: What are the signs that prompt injection is becoming a governance problem?

A: Warning signs include rising jailbreak success rates, tools being called in ways the business did not intend, and changes in behaviour after prompt or connector updates. If those patterns appear, the issue is no longer just model behaviour. It is a control failure across retrieval, access, and runtime policy.

Q: How should security teams govern AI models that can call tools and access data?

A: Security teams should govern AI models as non-human identities with named owners, limited scope, short-lived credentials, and continuous authorization. The critical shift is to treat every tool call, data read, and update path as a privileged action that can be logged, revalidated, and revoked. Without that discipline, model risk becomes identity risk.

Technical breakdown

Why deployed AI applications create a wider attack surface

A deployed AI application is not just a model endpoint. It is usually a chain of retrieval systems, prompts, external APIs, tool connectors, and business logic that turns text into action. That composition creates attack paths that do not exist in a standalone chatbot. Indirect prompt injection is especially important because malicious instructions can enter through a document, webpage, or API response that the system trusts as data. The model then processes those instructions in context and may follow them, even when the code itself is functioning as designed.

Practical implication: test the whole application path, including retrieval and tool execution, not just the model response layer.

Why AI red teaming differs from traditional pentesting

Traditional pentesting looks for deterministic flaws such as broken access control, misconfiguration, or unpatched code. AI red teaming adds a probabilistic layer because the same malicious input may succeed, fail, or partially succeed depending on prompt wording, context, and recent changes. That makes adversarial success a rate, not a binary outcome. It also means a team can have technically correct code and still be exposed if the model can be socially engineered into unsafe actions like data disclosure or unauthorised tool use.

Practical implication: measure repeatable attack success rates over many attempts, not only whether a single exploit worked.

How continuous testing tracks model and tool changes

AI systems change too quickly for annual testing to be sufficient. A new model version, a prompt rewrite, a new connector, or a fresh tool permission can reopen a path that was previously closed. Continuous AI red teaming keeps a baseline of adversarial attempts and reruns them as the application evolves, which is closer to how attackers operate in production. This also creates a governance signal: if prompt injection or tool-abuse success rates rise after a release, the issue is no longer theoretical, it is an operational regression.

Practical implication: tie adversarial retesting to every material model, prompt, or integration change before release.

Threat narrative

Attacker objective: The attacker wants to turn a trusted AI application into a tool for data exposure, unauthorised actions, or downstream system abuse.

1. Entry occurs when an attacker delivers malicious instructions through prompt injection, a retrieved document, or another external content source that the application ingests.
2. Escalation happens when the model treats that injected content as actionable context and is steered into calling tools, reading data it should not, or following hidden instructions.
3. Impact follows when the application leaks data, misuses connected systems, or performs a business action the attacker controlled through language rather than code exploit.

NHI Mgmt Group analysis

AI red teaming has become an identity and privilege problem, not only a model-safety problem. Once an LLM can call tools, query data, or act on behalf of users, its permissions behave like non-human identity entitlements. That means the security question is no longer only whether the model can be fooled, but whether the surrounding access model is narrow enough to limit damage. IAM and PAM teams should treat agentic workflows as governed identities with explicit scope, lifecycle, and revocation rules.

Indirect prompt injection is the named concept that security teams need to operationalise. The article shows that hidden instructions inside retrieved content can change application behavior without touching application code. That makes the control gap semantic trust, where systems cannot reliably distinguish data from instruction. Practitioners should read this as a sign that retrieval pipelines, external content sources, and tool outputs need security controls comparable to untrusted input handling elsewhere in the stack.

Continuous adversarial testing is now part of operational resilience for AI systems. Point-in-time assessments age quickly because model versions, prompts, and tool permissions change the application’s effective control boundary. That places AI red teaming in the same governance class as continuous validation in cloud and identity programmes. Security leaders should treat success-rate drift as a risk indicator, not a reporting metric.

Tool-connected AI creates governance debt when access reviews lag deployment. The more a model can reach email, databases, or file systems, the more the organisation depends on explicit permission scoping and revocation discipline. If those controls are not embedded early, the resulting blast radius is larger than the model team usually assumes. Practitioners should align AI rollout pace with access governance maturity, not with feature delivery alone.

OWASP-style application risk mapping is useful only if it is paired with identity controls. The article’s focus on prompt injection, tool misuse, and retrieval poisoning fits the application-layer view, but the real mitigation path includes credentials, service accounts, and delegated access. That intersection is where NHI governance matters most. Teams should model every AI connector as a controllable access path rather than a convenience feature.

What this signals

Semantic trust gap: enterprises are still building AI controls as if prompts, retrieved content, and tool calls were separate concerns. In practice, the security boundary is the full runtime path, which is why AI governance now overlaps with IAM, PAM, and NHI lifecycle controls. Teams that want a durable model should align testing with the OWASP Agentic AI Top 10 and MITRE ATT&CK-style attack-path thinking.

The operational signal for practitioners is clear. If model updates, prompt changes, or new connectors are not tied to mandatory retesting, control assurance will decay faster than policy can catch up. That is especially true when agents are allowed to act with standing permissions instead of short-lived, tightly scoped access.

For practitioners

• Map every AI application to its real attack surface Inventory retrieval sources, APIs, tool connectors, system prompts, and downstream systems before approval so red team coverage matches the deployed architecture.
• Treat AI tool access as governed privilege Assign the minimum possible permissions to service accounts and agent workflows, then define explicit revocation and rotation paths for every connected secret.
• Run adversarial tests on every material change Re-test after prompt edits, model swaps, new connectors, or tool permission changes because each of those can reopen an attack path that was previously closed.
• Track exploit success as an operational metric Measure the rate of successful jailbreaks, prompt injections, and unsafe tool calls over time, then use regressions to block release or force rework.

Key takeaways

• AI red teaming is shifting from model safety to runtime governance because the deployed application, not the model alone, determines the attack surface.
• Continuous testing matters because prompt changes, new tool access, and model updates can reopen attack paths that a previous assessment had already closed.
• IAM, PAM, and NHI controls are now part of AI security whenever agents can retrieve data or call tools on behalf of the business.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Indirect Prompt Injection: Indirect prompt injection is an attack where malicious instructions are hidden inside content that an AI system reads later. The model may treat that content as context rather than as hostile input, which can influence tool use, data access, or workflow actions if controls are weak.
• Agentic workflow: An agentic workflow is a sequence of tasks executed by an AI agent with some level of tool access and decision authority. In security terms, the workflow matters because it can span multiple systems, identities, and permissions, which makes attribution and revocation harder than with ordinary automation.
• Semantic Data Exfiltration: Semantic data exfiltration is the leaking of sensitive information through natural-language interaction rather than a conventional software exploit. In AI systems, an attacker may coax the model into revealing data it can access, even when the underlying code and infrastructure appear intact.

What's in the full article

Novee's full research covers the operational detail this post intentionally leaves for the source:

• Step-by-step red teaming workflows for LLM-powered applications, including scoping, probing, validating, and reporting.
• Examples of prompt injection, jailbreak, and tool-abuse payloads that can be adapted to different application architectures.
• How automation and human operators combine to test agentic workflows and track regression over time.
• The source article's practical mapping of AI red teaming to OWASP LLM application risks and enterprise deployment patterns.

👉 Novee’s full article covers the testing workflow, attack patterns, and continuous validation approach in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, IAM, human identity, identity lifecycle, and secrets management. It helps practitioners connect AI adoption to the access controls their programmes already own.