AI coding agents: what they miss in real red teaming

TL;DR: Pointing Claude Code, Cursor, or Codex CLI at an AI application produces only an orchestrator without the harness needed for real red teaming, according to Repello AI, because refusal-free payload generation, attack memory, verification, and coverage chaining are all missing. That distinction matters because assurance depends on proving exploit success, not producing a report-shaped artifact.

Editorial analysis by NHI Mgmt Group, based on content published by Repello AI: “Claude Code Red Teaming: What a Coding Agent Can't Do”.

Key questions

Q: What breaks when a coding agent is used as a red team without a harness?

A: The test loses the four things that make red teaming credible: authorised exploit generation, memory of what worked, independent verification, and chained coverage across surfaces.

Q: Why do frontier models refuse to generate working jailbreaks or exfiltration payloads?

A: Because they are trained and filtered to avoid harmful dual-use content, which is correct for an assistant but limiting for adversarial testing.

Q: How do security teams know if AI red teaming is working?

A: AI red teaming is working when testing finds real prompt injection paths, over-scoped integrations, and policy gaps before attackers do, and when fixes are re-tested successfully after model or workflow changes.

Practitioner guidance

• Define the testing boundary before you test Separate reconnaissance, exploit generation, and assurance into different operating modes so a one-shot assistant is never mistaken for a red-team harness.
• Require independent verification of every claimed finding Use a verifier that is distinct from the attacker so the system can confirm whether a payload actually exfiltrated data, executed a tool call, or only looked plausible.
• Build a reusable payload library Store confirmed attack chains, the conditions that made them work, and the guardrails they bypassed so later tests can adapt rather than restart from zero.

Bottom line: A coding agent pointed at an AI app can be useful for exploration, but it does not become a red team until the surrounding system can generate, verify, remember, and chain attacks.

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →