Join our Newsletter — 33% off our NHI Course

Notifications
Clear all

Agentic coding at scale: are your review and eval loops ready?


(@nhi-mgmt-group)
Member Moderator
Joined: 1 year ago
Posts: 18936
Topic starter  

TL;DR: Agentic coding has made code generation cheap enough that review, testing, and traceability are now the scarce controls, according to Arize’s field report from AI Engineer Europe. The practical consequence is that engineering quality is becoming a governance problem, because unverified AI-generated changes can outpace human oversight and quietly raise operational and security risk.

NHIMG editorial — based on content published by Arize: Code is free, technical debt isn’t: Notes from AI Engineer Europe

Questions worth separating out

Q: How should security teams govern AI-generated identity workflows in application code?

A: Treat them as controlled code changes, not convenience scaffolding.

Q: When does agentic coding become a security risk rather than a productivity gain?

A: It becomes risky when generation outpaces verification.

Q: What do teams get wrong about securing AI coding assistants?

A: Teams often focus on code output and ignore the agent boundary, where file reads, tool outputs, and external content shape the next action.

Practitioner guidance

  • Define review gates for AI-generated code Require human approval for changes that affect authentication, authorization, secrets handling, or workflow permissions, and set tighter thresholds for large or repetitive agent-produced diffs.
  • Build eval suites for policy-bearing code Create regression tests that validate access decisions, session handling, and secret-use patterns before changes merge into production.
  • Instrument agent traces end to end Capture prompts, sub-agent steps, tool calls, and outputs so failures in production can be tied back to the exact decision point.

What's in the full article

Arize's full field report covers the operational detail this post intentionally leaves for the source:

  • Session-level examples of how teams are using evals, traces, and test harnesses to catch agent-generated defects before release
  • Practitioner commentary on where human review still adds value when model output is already fast enough to overwhelm normal workflows
  • Detailed examples of coding agent failures and the verification patterns used to isolate which step broke
  • Conference-based observations on how teams are reorganising around review capacity, quality, and human judgment

👉 Read Arize's field report on AI Engineer Europe and agentic coding verification →

Agentic coding at scale: are your review and eval loops ready?

Explore further

View Full Forum →  |  NHI Foundation Course →



   
Quote
(@mr-nhi)
Member Moderator
Joined: 3 months ago
Posts: 18527
 

Verification debt is now a governance issue, not just an engineering issue. When code generation becomes cheap, the limiting factor moves to proof, review, and control. That is the same logic identity teams already apply to privileged access, where speed without oversight creates blast radius. The practical conclusion is that AI-assisted delivery should be governed as a control environment, not only as a productivity layer.

A question worth separating out:

Q: How can organisations prove their AI controls are actually working?

A: Look for evidence that policy decisions are logged, sensitive prompts are being redacted or blocked when required, and approved AI interactions are traceable by identity and business context. Effective programmes produce audit-ready records, not just policy text. If the control cannot explain what happened in a session, it is not operational enough.

👉 Read our full editorial: Agentic coding has shifted the bottleneck from output to verification



   
ReplyQuote
Share: