Join our Newsletter — 33% off our NHI Course

Could Claude Code cover real AI red teaming without a harness?

 

(@nhi-mgmt-group)
Member Moderator
Joined: 2 years ago
Posts: 20225
Topic starter  

TL;DR: The harness needed for real AI red teaming is still missing when Claude Code, Cursor, or Codex CLI are aimed at an application, Repello AI argues, even if the agent helps with exploration. Without generation, verification, memory, and chained coverage, findings stay provisional.

Editorial analysis by NHI Mgmt Group, based on content published by Repello AI: “Claude Code Red Teaming: What a Coding Agent Can't Do”.

Key questions

Q: Why do frontier models refuse to generate working jailbreaks or exfiltration payloads?

A: Because they are trained and filtered to avoid harmful dual-use content, which is correct for an assistant but limiting for adversarial testing.

Q: How do security teams know if AI red teaming is working?

A: AI red teaming is working when testing finds real prompt injection paths, over-scoped integrations, and policy gaps before attackers do, and when fixes are re-tested successfully after model or workflow changes.

Q: What breaks when a coding agent is used as a red team without a harness?

A: The test loses the four things that make red teaming credible: authorised exploit generation, memory of what worked, independent verification, and chained coverage across surfaces.

Practitioner guidance

  • Define the testing boundary before you test Separate reconnaissance, exploit generation, and assurance into different operating modes so a one-shot assistant is never mistaken for a red-team harness.
  • Require independent verification of every claimed finding Use a verifier that is distinct from the attacker so the system can confirm whether a payload actually exfiltrated data, executed a tool call, or only looked plausible.
  • Build a reusable payload library Store confirmed attack chains, the conditions that made them work, and the guardrails they bypassed so later tests can adapt rather than restart from zero.

Bottom line: A coding agent pointed at an AI app can be useful for exploration, but it does not become a red team until the surrounding system can generate, verify, remember, and chain attacks.

Glossary terms: AI Red Teaming, Verifier Loop, Coverage Engine, Delegated AI Authority

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →


This topic was modified 1 week ago by NHI Mgmt Group

   
Quote
(@mr-nhi)
Member Moderator
Joined: 6 months ago
Posts: 20715
 

AI red teaming without a harness is category error, not capability. A frontier model can orchestrate probes, but orchestration alone does not create assurance. The missing elements are authorised exploit generation, persistent attack memory, independent verification, and coverage across surfaces. Practitioners should treat one-pass agent testing as reconnaissance, not as evidence that an AI system is hardened.

A few things that frame the scale:

  • Claude Code-assisted commits leaked secrets at a rate of 3.2%, more than double the human-only baseline of 1.5%, with peaks reaching 31 secrets per 1,000 commits in August 2025, according to the State of Secrets Sprawl 2026.

A question worth separating out:

Q: What should teams do when AI systems can act on behalf of the enterprise?

A: Treat those systems as non-human identities with explicit scope, traceable actions, and fast revocation paths. The practical test is whether the organisation can see what the agent did, restrict what it may do next, and stop it without rebuilding the workflow from scratch.

👉 Read our full editorial: Why coding agents alone are not enough for AI red teaming


This post was modified 1 week ago by NHI Mgmt Group

   
ReplyQuote
Share:

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.