Why coding agents alone are not enough for AI red teaming

At a glance

What this is: This analysis says a coding agent pointed at an AI app is not a real red teamer, because it lacks the harness needed to generate working attacks, retain attack memory, verify success, and chain coverage across surfaces.

Why it matters: For IAM, NHI, and AI security teams, the issue is governance over testing capability: assurance changes when you can prove exploitability, not just observe model output.

Context

AI red teaming is often confused with a single capable model running probes against one surface, but that is only a narrow slice of adversarial testing. Real assurance needs a harness that can generate working attacks under authorised conditions, remember what succeeded, verify impact independently, and test beyond the first surface that was pointed at the target.

Where AI systems can call tools, read retrieved content, and act across multiple turns, the testing problem starts to resemble identity and privilege governance as much as model evaluation. Red Teaming AI Agents for Identity Abuse is a useful reference point because the article's core argument is about controlled exploitation, delegation, and abuse of runtime authority rather than prompt quality alone.

Key questions

Q: What breaks when a coding agent is used as a red team without a harness?

A: The test loses the four things that make red teaming credible: authorised exploit generation, memory of what worked, independent verification, and chained coverage across surfaces. The result is usually a report-shaped artifact, not proof that the application resisted a determined attack. That is useful for exploration, but it is not assurance.

Q: Why do frontier models refuse to generate working jailbreaks or exfiltration payloads?

A: Because they are trained and filtered to avoid harmful dual-use content, which is correct for an assistant but limiting for adversarial testing. A red-team workflow needs an authorised context where producing the working payload is the purpose, not a misuse case. Otherwise the model will soften, refuse, or sanitise the attack.

Q: How do security teams know if AI red teaming is working?

A: AI red teaming is working when testing finds real prompt injection paths, over-scoped integrations, and policy gaps before attackers do, and when fixes are re-tested successfully after model or workflow changes. The strongest signal is repeatable reduction in exposed authority, not a lower number of red-team findings on its own.

Q: What should teams do when AI systems can act on behalf of the enterprise?

A: Treat those systems as non-human identities with explicit scope, traceable actions, and fast revocation paths. The practical test is whether the organisation can see what the agent did, restrict what it may do next, and stop it without rebuilding the workflow from scratch.

Technical breakdown

Why a frontier model is only the orchestrator

A frontier model can plan, reason across steps, call tools, and write code, which makes it a strong orchestrator. But orchestration is not the same as a red-team system. The article's central distinction is that the model can decide what to try next, yet it still lacks the scaffolding that turns those choices into repeatable assurance. In practice, the model supplies the reasoning core, while the harness supplies the policy, persistence, and instrumentation needed to make adversarial testing reliable. Without that outer system, you get a capable one-pass assistant, not a campaign that can prove failure modes across a changing application surface.

Practical implication: Treat the model as a component inside a testing workflow, not as the workflow itself.

Why refusal systems cap exploit generation

General-purpose frontier models are trained to avoid producing harmful dual-use content, using reinforcement learning from human feedback, constitutional policies, and layered safety classifiers. That makes sense for an assistant, but it creates a ceiling for red teaming because the model may soften, refuse, or sanitize the very payloads the test must produce. The article argues that a red-team harness solves this by operating in an authorised context where working exploits are the point of the exercise, not an accidental misuse case. This is the difference between asking for a safe illustration of an attack and getting a payload that actually lands.

Practical implication: Run exploit generation inside an authorised harness, not through a general assistant prompt.

How memory, verifier loops, and coverage change the result

Real red teaming is cumulative. A payload that works once becomes part of a library, a verifier confirms whether the objective was actually met, and the coverage engine expands testing across turns, surfaces, and modalities. That feedback loop is what turns isolated findings into repeatable assurance. Without it, a model either grades its own homework or stops at the first surface it was shown. The article also notes that AI applications expose multiple attack planes, including prompt injection, tool boundaries, retrieval pipelines, and multimodal inputs, so point tests miss the chain where the exploit actually lives.

Practical implication: Build campaigns that store confirmed payloads, verify outcomes independently, and retest across all exposed surfaces.

Threat narrative

Attacker objective: The objective is to produce a credible finding or proof of exploitability, but the bare coding agent cannot reliably reach that goal without the missing harness functions.

1. Entry begins when a user points a capable coding agent at an AI application and asks it to red team the surface without a broader harness.
2. Escalation occurs when the agent probes one surface, but the lack of refusal-free context, attack memory, and verifier logic limits how far the campaign can go.
3. Impact is a false sense of assurance, because the output looks like a red-team report while untested chains, missed surfaces, and unverified findings remain in place.

Breaches seen in the wild

• Sentry MCP Agentjacking 2026: Researchers showed a fake Sentry error, posted with a public DSN, could make AI coding agents run attacker code with developers' credentials.
• Nx s1ngularity attack 2025: Attackers stole Nx's npm token via a GitHub Actions flaw and shipped malware that stole 2,349 secrets and abused developers' AI CLIs.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI red teaming without a harness is category error, not capability. A frontier model can orchestrate probes, but orchestration alone does not create assurance. The missing elements are authorised exploit generation, persistent attack memory, independent verification, and coverage across surfaces. Practitioners should treat one-pass agent testing as reconnaissance, not as evidence that an AI system is hardened.

Refusal behaviour is a governance feature for assistants and a constraint for red teams. The same safety layer that protects general users also prevents a bare model from reliably producing working jailbreaks or exfiltration chains. That means the test environment must supply the authorisation boundary, not the model. For identity-heavy AI systems, this is where runtime authority becomes the real subject of evaluation, not prompt wording alone.

Coverage gaps are the hidden failure mode in AI security testing. Multi-turn jailbreaks, indirect prompt injection, tool calls, retrieval poisoning, and multimodal inputs create chains that a one-shot agent will not naturally traverse. In governance terms, the failure is not lack of ideas but lack of chained proof. Security teams should assume that single-surface testing leaves exploitable paths undiscovered.

Attack memory is what separates repeatable red teaming from ad hoc probing. A harness that learns from prior successes turns testing into an accumulating control function, while stateless prompting resets every session. That matters because AI applications change quickly, and the same target class can fail in the same way again. Teams should regard evolving payload libraries as an assurance asset, not a convenience.

Red Teaming AI Agents for Identity Abuse is the right lens when AI systems can act with delegated authority. Once an application can read context, use tools, and make decisions across turns, it begins to resemble an identity-bearing actor in the attack path. That means red-team quality is inseparable from privilege scope, delegation boundaries, and proof of action. Practitioners should evaluate the testing stack the same way they evaluate any privileged runtime: by what it can actually do, not by how it describes itself.

From our research library:

• Claude Code-assisted commits leaked secrets at a rate of 3.2%, more than double the human-only baseline of 1.5%, with peaks reaching 31 secrets per 1,000 commits in August 2025, according to the State of Secrets Sprawl 2026.
• Read next: AI Coding Agents Security Guide

What this signals

Attack success has to be measured at the system level, not the prompt level. The article's core warning is that a model can generate fluent probes while still failing to produce a working exploit chain, which is why verification has to sit outside the attacker. That distinction becomes more important as AI systems expose tool use, retrieval, and delegated execution in the same runtime.

AI red teaming becomes a governance control when attack memory is persistent. A reusable library of confirmed payloads turns each engagement into input for the next one, which is how testing starts to resemble a living control rather than a one-off event. In adjacent research, Claude Code-assisted commits leaked secrets at a rate of 3.2%, more than double the human-only baseline of 1.5%, with peaks reaching 31 secrets per 1,000 commits in August 2025, according to the State of Secrets Sprawl 2026.

Delegated AI authority changes the security boundary. Once an AI system can act on retrieved content and use tools, the governance problem shifts from output quality to who or what is allowed to act. That makes harness-based red teaming a prerequisite for any programme that treats AI behaviour as operationally relevant, especially where task-scoped and just-in-time access would reduce blast radius.

For practitioners

• Define the testing boundary before you test Separate reconnaissance, exploit generation, and assurance into different operating modes so a one-shot assistant is never mistaken for a red-team harness.
• Require independent verification of every claimed finding Use a verifier that is distinct from the attacker so the system can confirm whether a payload actually exfiltrated data, executed a tool call, or only looked plausible.
• Build a reusable payload library Store confirmed attack chains, the conditions that made them work, and the guardrails they bypassed so later tests can adapt rather than restart from zero.
• Map coverage across all exposed surfaces Include prompt injection, tool and MCP boundaries, retrieval pipelines, and multimodal inputs in scope so testing follows the real attack chain instead of the first surface presented.

Key takeaways

• A coding agent pointed at an AI app can be useful for exploration, but it does not become a red team until the surrounding system can generate, verify, remember, and chain attacks.
• The article's central warning is governance-related: if the workflow cannot prove exploit success, the output is only a plausible finding, not assurance.
• Practitioners should evaluate AI red teaming as a system of controls, with coverage, verification, and persistent attack memory treated as core capabilities rather than optional extras.

Related NHI Mgmt Group resources: Red Teaming AI Agents for Identity Abuse · AI Coding Agents Security Guide · AI Red Teaming Guide · Sentry MCP Agentjacking 2026

Key terms

• AI Red Teaming: AI red teaming is the practice of simulating hostile behaviour against models, applications, and agents to expose weaknesses before real attackers do. In AI programmes, it is most useful when results can be turned into controls, monitoring, and governance evidence rather than left as a one-time test report.
• Verifier Loop: A verifier loop is a repeated check that confirms an identity, action, or system state before trust is granted. In identity security, it combines policy evaluation, token or credential validation, and contextual signals to continually reassess NHI, human, or agent access across sessions, workflows, and tool use.
• Coverage Engine: A coverage engine is the part of an adversarial testing system that ensures the target is examined across all relevant surfaces, turns, and modalities. It prevents single-surface bias by chaining tests across prompt injection, tool use, retrieval paths, and other execution surfaces.
• Delegated AI Authority: Delegated AI authority is the permission a person gives an assistant to act inside business systems on their behalf. It turns a conversational tool into an execution layer, which means security teams must govern scope, auditability, and revocation with the same seriousness they apply to privileged access.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect delegated authority, runtime access, and identity risk across modern security programmes.