AI red teaming is now a governance issue for LLM applications

At a glance

What this is: Novee argues that AI red teaming for LLM applications must assess the live, connected system, because the main risks come from prompt injection, tool abuse, and changing application behavior.

Why it matters: For IAM and NHI practitioners, the key issue is that connected tools, API access, and agentic workflows behave like governed access paths, so testing has to cover privilege, inputs, and execution together.

Context

AI red teaming for LLM applications is a governance problem because the attack surface is the deployed system, not the model alone. Once an LLM can read retrieval content, call tools, or act on behalf of a workflow, the security question shifts from output quality to whether adversarial input can trigger real business actions.

That creates a direct identity and access angle for IAM, PAM, and NHI teams. Connected tools, API integrations, and agentic workflows effectively extend the application’s authority, so their permissions, trust boundaries, and validation rules need to be treated as part of the control plane. The article’s starting position is typical of mature AI security programmes, but it still catches many organisations off guard.

Key questions

Q: What breaks when an LLM application is tested only as a model?

A: Model-only testing misses the real attack surface created by retrieval, tool access, and business logic. A model may refuse harmful prompts in a chat box and still be steered into leaking data or misusing a connected tool once it is deployed inside a live application. The failure is architectural, because authority exists outside the model.

Q: Why do connected tools turn prompt injection into a governance problem?

A: Because the injected instruction is no longer just text. It becomes a way to consume real authority if the application can reach email, databases, files, or APIs. The governance issue is whether the delegated access is bounded tightly enough that a malicious prompt cannot convert language into action.

Q: How do security teams know if AI red teaming is working?

A: AI red teaming is working when testing finds real prompt injection paths, over-scoped integrations, and policy gaps before attackers do, and when fixes are re-tested successfully after model or workflow changes. The strongest signal is repeatable reduction in exposed authority, not a lower number of red-team findings on its own.

Q: How should organisations govern agentic AI in observability workflows?

A: Treat AI as a bounded operator, not an all-purpose controller. Give it narrow permissions, require audit trails, and restrict it to approved actions such as validation, suggestion, or rollback of pre-defined changes. If the telemetry path is fragmented, fix that first, because agentic automation amplifies whatever control model already exists.

Technical breakdown

Why deployed LLM apps fail differently from model-only testing

An LLM model in isolation can be tested for bias, toxicity, or refusal behavior, but a deployed application inherits everything around it: retrieval sources, tool permissions, orchestration logic, and business rules. That means an attacker does not need to break the model. They only need to steer the application into using legitimate capabilities in unsafe ways. Indirect prompt injection is the clearest example, because the malicious instruction can arrive through a document or API response that the application trusts as data. The risk is architectural, not just linguistic, which is why model-only safety testing misses whole classes of failure.

Practical implication: Test the full application path, including data sources and tool calls, not just the model prompt.

Prompt injection and tool misuse as access-control failures

Once an LLM can reach email, databases, or file systems, the control problem looks less like content moderation and more like privilege governance. The model may be able to issue a valid command, but that does not mean it should. In practice, prompt injection becomes a way to coerce an application into misusing a connected tool, leaking accessible data, or following hidden instructions embedded in content. For identity teams, that means the question is not whether the model can generate the right text. The question is whether delegated authority is bounded tightly enough to resist manipulation.

Practical implication: Treat every tool-enabled LLM workflow as a privilege-bearing identity and restrict its effective scope.

Why continuous testing is required as AI applications change

Point-in-time red teaming ages quickly because the attack surface changes whenever the provider updates a model, engineers revise a system prompt, or a new tool is added. A defense that held last week may fail after a silent model change or a new integration widens the reachable actions. That is why continuous testing is more than an audit preference. It is the only way to track whether adversarial success rates are improving or deteriorating over time. In operational terms, AI security becomes a regression problem, not a one-off assessment.

Practical implication: Re-run adversarial tests whenever prompts, tools, or models change, and track success rates as a release metric.

Threat narrative

Attacker objective: The attacker wants to coerce the deployed LLM application into revealing data or performing actions through its own legitimate integrations.

1. Entry begins when a malicious instruction is hidden inside retrieved content, an external data source, or a crafted user prompt.
2. Credential or capability abuse follows when the LLM application is steered into using its connected tools or data access in ways the operator did not intend.
3. Impact occurs when the application leaks data, performs an unauthorized action, or follows attacker-controlled instructions in production.

Breaches seen in the wild

• ForcedLeak (Salesforce Agentforce) 2025: A critical Agentforce flaw let attackers steer an AI agent through a Web-to-Lead form to leak CRM data via an expired allowlisted domain.
• Microsoft Azure OpenAI abuse by Storm-2139: Storm-2139 used API keys leaked by Microsoft customers to hijack Azure OpenAI, bypass safety guardrails and resell access to generate harmful content.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM red teaming has become an access-governance discipline, not just an AI safety exercise. The article is right to separate model output testing from application testing, because the real risk appears when a model is connected to tools, retrieval, and business logic. That makes the control question one of delegated authority, not simply unsafe text generation. Practitioners should govern the application like any other privilege-bearing system.

Indirect prompt injection is a trust-boundary failure, not a content problem. The article shows that hidden instructions inside documents or API responses can hijack an application even when the code path is functioning normally. That is a governance gap because the system is treating untrusted content as actionable instruction. Practitioners should classify retrieval and external inputs as adversary-controlled until proven otherwise.

Continuous red teaming exposes AI governance debt. When model versions, prompts, and tool access change faster than review cycles, yesterday’s assessment no longer describes today’s attack surface. This creates a named concept worth tracking: AI governance debt: the growing gap between how fast an AI application changes and how slowly its controls are revalidated. Practitioners should measure that gap directly, not assume annual testing is enough.

Agentic workflows must be governed as identities with least privilege. The article’s own framework alignment to the OWASP Top 10 for Agentic Applications is appropriate because tool-using agents can accumulate excess authority through integrations. That is where IAM and NHI teams matter most: the workflow is not just software, it is a runtime actor with reach. Practitioners should bound tool access, not just prompt content.

The strongest programmes will merge AI red teaming with application security testing. Traditional pentesting still matters for APIs and authorization flaws, but it does not reveal the same behavioral weaknesses that language models introduce. The field is moving toward dual coverage because attackers chain model steering with ordinary application defects. Practitioners should test both the language layer and the connected systems together.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Identity Security Buyer's Guide

What this signals

AI governance debt: the fastest-growing failure mode is not a bad model, but a mismatch between how quickly an LLM application changes and how slowly its controls are retested. That gap widens every time a team adds a tool, rewrites a prompt, or accepts a silent model update, so the practical response is release-triggered adversarial testing rather than annual review.

AI red teaming is increasingly an IAM and NHI issue because connected tools behave like delegated credentials in motion. The programme question is no longer whether the model can be tricked in theory, but whether the runtime actor can be prevented from turning untrusted content into authorised action.

AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026. That pattern reinforces the need to govern the broader application stack, not just the model layer.

For practitioners

• Map every connected tool and data source Inventory retrieval pipelines, API integrations, file stores, and workflow steps that an LLM application can reach, then classify which actions are read-only, write-capable, or externally visible.
• Limit delegated authority for agentic workflows Assign the narrowest feasible tool permissions, separate high-risk actions from low-risk ones, and require explicit approval for actions that alter records, send messages, or move data.
• Test for indirect prompt injection Include malicious instructions embedded in retrieved documents, web pages, and API responses so you can see whether the application treats untrusted content as commands.
• Measure adversarial success over time Track the share of successful jailbreaks, tool misuse attempts, and data-exfiltration probes across model versions and prompt revisions, then use the trend as a release gate.
• Retest after every material change Re-run the same adversarial suite whenever you change the system prompt, add a tool, update a model, or alter business logic that influences the assistant’s actions.

Key takeaways

• LLM red teaming has to examine the deployed application, because the real exposure comes from connected tools, retrieval, and runtime authority, not from model output alone.
• Prompt injection becomes materially more dangerous when an assistant can read, write, or trigger actions in business systems, which makes delegation boundaries a core governance issue.
• Continuous testing is necessary because model updates, prompt edits, and new integrations can reopen attack paths that looked closed during the last assessment.

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Indirect Prompt Injection: Indirect prompt injection is an attack where malicious instructions are hidden inside content that an AI system reads later. The model may treat that content as context rather than as hostile input, which can influence tool use, data access, or workflow actions if controls are weak.
• Tool Misuse: Tool misuse occurs when an agent uses an allowed integration in a way that exceeds its intended task, scope, or risk tolerance. The problem is often not access alone but the combination of valid credentials, broad permissions, and unbounded action sequencing.
• Agentic workflow: An agentic workflow is a sequence of tasks executed by an AI agent with some level of tool access and decision authority. In security terms, the workflow matters because it can span multiple systems, identities, and permissions, which makes attribution and revocation harder than with ordinary automation.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is suitable for teams that need to connect access control, delegated authority, and runtime governance.