Promptware exposes why static AI red teaming misses agentic attacks

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Preamble's blog post: Static AI red teaming fails against Promptware because agentic systems can be manipulated across tool use, memory, and execution stages, not just at the prompt boundary.

Why it matters: Practitioners need to test AI systems as operational environments with privileges, dependencies, and memory, because a single unsafe prompt is rarely the full attack path.

Context

AI red teaming breaks down when it assumes the only risk is a bad prompt response. Agentic systems with tools, retrieval, memory, and execution paths create a wider attack surface, so the security question shifts from prompt hygiene to control over what the system can do after it accepts input.

The application layer is where deployers own the risk, especially when an AI system reaches into internal APIs, databases, or code execution. That makes identity, authorization, and least privilege part of AI security, not a separate concern.

The article's starting position is not atypical for enterprise AI deployments, because many organisations still assess these systems as if they were chat interfaces rather than operational actors.

Key questions

Q: How do prompt injections affect agentic workflows?

A: Prompt injection can manipulate an AI agent’s instructions so it follows attacker-supplied text instead of its intended policy, which can lead it to call tools or export outputs with unauthorised data. The main controls are prompt isolation, strict tool allowlisting, output filtering, human approval for high-risk actions, and runtime monitoring of the agent’s delegated authority.

Q: Why does excessive agency in an AI system create such a broad security risk?

A: Excessive agency is risky because it expands what the model can do, not just what it can say. If an LLM can reach backend systems, use broad credentials, or execute privileged functions, a prompt injection or jailbreak can turn into unauthorized access, data leakage, destructive changes, or remote execution. The larger the action surface, the larger the blast radius.

Q: What is the difference between prompt injection and jailbreaking in AI security?

A: Jailbreaking targets the model’s safety rules and tries to make it produce disallowed content. Prompt injection targets the application’s instructions, context, tools, and policies, and can trigger data leakage or wrong actions even if the model itself is behaving normally. Security teams should test for both because they fail in different places.

Q: What are the signs that an AI agent attack is only becoming visible after compromise has already happened?

A: Look for a legitimate-looking tool call, process spawn, identity use or outbound connection that appears normal in isolation but fits an unusual sequence, scope or rate for that agent. The visible event is the clue that earlier silent stages may already have shaped the task.

Technical breakdown

Why static jailbreak tests miss agentic AI risk

Static jailbreak testing focuses on whether a model will emit disallowed content when prompted directly. That works only for a narrow interaction model. Agentic AI changes the attack surface because the system can call tools, retrieve data, write to memory, and trigger actions across multiple steps. The attacker does not need to win at the first prompt if they can steer the system into a later stage where the model has already been trusted and equipped with permissions. This is why point-in-time prompt libraries are a poor proxy for real exposure in operational deployments. See the Agentic AI Security Guide.

Practical implication: Test the full tool graph and stateful workflows, not just chat prompts.

How the Promptware kill chain advances through persistence and C2

The Promptware kill chain models a campaign rather than a single exploit. Initial access may come from injected text, poisoned documents, or retrieved web content. From there, the attacker seeks privilege escalation, reconnaissance, persistence in memory or retrieval stores, and command-and-control through recurring retrieval of hostile content. That matters because a vulnerable AI system can be re-steered after the first interaction ends. The campaign structure makes prompt injection look less like a one-off input validation failure and more like an ongoing control problem across the system's state and dependencies. See the Cyber Kill Chain and Diamond Model Guide.

Practical implication: Instrument memory, retrieval, and tool access as persistence surfaces that need continuous monitoring.

Why excessive agency creates an identity problem

Excessive agency means the AI can use the privileges it was granted to complete tasks that go beyond what the operator intended. In practice, that turns an agent into a pre-authenticated insider with access to email, data stores, code, or financial systems. The governance issue is not the model's intelligence alone, but the scope of delegated authority attached to it. Once an agent can act across systems, identity, access scope, and transaction boundaries become the controls that determine whether a prompt becomes an incident. See the Agentic AI Identity Guide.

Practical implication: Bind every agent to least-privilege access and review delegated authority as a first-class control.

NHI Mgmt Group analysis

Static prompt testing is the wrong control plane for agentic AI. A scanner that only checks whether an assistant will say something unsafe tells you little about whether it will misuse tools, modify records, or persist malicious state. The deeper issue is that the attack surface now includes execution context, not just model output. Practitioners should treat prompt libraries as a narrow input test, not as a complete red team result.

Promptware turns AI security into a lifecycle problem, not a single-test problem. Once malicious instructions can survive in memory, retrieval corpora, or multi-turn dialogue, the relevant question is no longer whether the model rejected one prompt. It is whether the system can be re-influenced after the first interaction and whether its stored state can be poisoned. That makes state management part of the security model, especially in systems that mix retrieval, tools, and long-running sessions.

Excessive agency is the named failure mode behind many agentic AI incidents. The article's most useful concept is not prompt injection alone but the way delegated authority expands the blast radius of a successful injection. An agent with broad access becomes an operational insider with valid credentials and legitimate pathways into business systems. The control failure is over-delegation, so practitioners should constrain agent authority before testing for clever prompts.

The market is still measuring the wrong thing. Most organisations celebrate a clean result against a static jailbreak corpus, but that says little about adversarial behavior across tool use, persistence, and lateral movement. This is why the field needs kill-chain-aware evaluation as a baseline rather than an advanced capability. Security teams should judge AI red teaming by whether it exposes the full campaign path, not whether it catches offensive wording.

From our research library:

• 67% of organisations still rely heavily on static credentials despite the risks they pose to agentic AI deployments, according to the 2026 Infrastructure Identity Survey.
• Read next: Agentic AI Identity Guide

What this signals

Promptware is forcing a governance reset: 69% of security leaders agree identity management must fundamentally shift to address agentic AI systems, according to the 2026 Infrastructure Identity Survey. That shift is not about naming the model as a user, but about controlling how delegated authority is issued, scoped, and revoked across the workflow.

Context-aware red teaming will become the baseline: static prompt tests will keep missing persistence, tool misuse, and cross-system propagation. Security teams should expect evaluation programmes to move toward campaign-style testing that mirrors how agents actually operate in production.

Static credentials remain a structural weakness: 67% of organisations still rely heavily on static credentials despite the risks they pose to agentic AI deployments, according to the 2026 Infrastructure Identity Survey. Agentic systems that depend on long-lived secrets will keep expanding blast radius unless issuance and revocation become part of the control design.

For practitioners

• Map agent permissions to actual business impact Inventory every tool, API, data source, and execution path an agent can reach, then classify the business consequence if each one is misused.
• Replace static jailbreak corpora with kill-chain scenarios Build red team cases that cover initial access, escalation, persistence, retrieval-based command and control, and actions on objective.
• Treat retrieval and memory as hostile input surfaces Sanitise documents, web content, and vector-store entries before they can influence downstream decisions or survive across sessions.
• Constrain delegated authority to least privilege Remove unnecessary tool access, narrow write permissions, and separate read, act, and approval functions wherever the workflow allows.

Key takeaways

• Promptware shows that AI red teaming must assess multi-stage attack paths, not just whether a model produces unsafe text.
• Agentic AI becomes an identity problem as soon as delegated tools, memory, and execution rights create a real blast radius.
• Practitioners should move from static jailbreak libraries to kill-chain testing that measures persistence, propagation, and downstream impact.

Related NHI Mgmt Group resources: Agentic AI Identity Guide · Agentic AI Security Guide · Cyber Kill Chain and Diamond Model Guide · Red Teaming AI Agents for Identity Abuse

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Jailbreak: A prompt or input transformation that bypasses a language model’s safety restrictions and causes it to produce output it would normally refuse. In operational settings, a jailbreak matters because the model may be embedded in a workflow, making the bypass a pathway to broader misuse, not just a bad response.
• Excessive agency: A condition where an AI system is given more operational authority than its task requires. The risk is not just poor output. It is that mistakes, manipulation, or compromise can produce destructive actions at machine speed across the systems the agent can reach.
• Retrieval-augmented Generation: Retrieval-augmented generation is a pattern where an AI model pulls external information before generating output. The security challenge is that access rules can weaken when data is chunked, embedded, cached, or reused, so source permissions may not automatically follow the content into the model's context.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, and secrets management. It helps practitioners connect identity controls to the systems that now act, decide, and persist across workflows.