Coding agents need telemetry to support self-verifying software loops

At a glance

What this is: This is an analysis of why coding agents need telemetry, trace access, and evaluation loops to improve software reliably at runtime.

Why it matters: It matters because IAM, NHI, and AI security teams must govern agent access, observability, and verification boundaries when software decisions shift from static code to runtime behaviour.

By the numbers:

• A recent large-scale study of GitHub repositories estimated that 16 to 23 percent of code contributions already involve coding agents.

👉 Read Arize's analysis of coding agents, telemetry, and self-improving software loops

Context

Coding agents are changing how software is written, tested, and debugged, but the governance gap is that many teams still assume deterministic code review and human verification are enough. In practice, the real control surface has moved into the runtime loop, where the agent selects tools, reuses context, and changes behaviour based on telemetry. That creates an identity and access question for the software itself, because the agent is not just a model call, it is a system that needs governed tool access and traceability.

The first-order issue is not whether agents can generate code, but whether they can verify that code against evidence from the environment they operate in. That shifts security, engineering, and IAM teams toward observability, scoped permissions, and auditability for AI-driven workflows. It also means the quality of the harness matters as much as the model, because access without verification increases the chance of silent failure at scale.

Key questions

Q: How should teams govern AI-assisted development workflows that use coding agents?

A: Treat them as identity-governed execution paths, not just productivity tools. Define who can start the workflow, which tools and data sources it can reach, what evidence is required for review, and how access is revoked if the workflow expands beyond its intended scope. The key is to govern the chain of delegated action, not only the final code output.

Q: Why do traces matter so much for agentic software?

A: Because in agentic systems, traces describe what the system actually did at runtime. Code shows intended structure, but traces reveal tool use, looping, retries, and failure points. Without trace access, neither the agent nor the reviewer can reliably prove whether a change improved behaviour or just looked correct.

Q: What breaks when coding agents do not have governed observability?

A: Verification breaks first, then debugging, then trust in the agent’s output. If telemetry is missing, noisy, or inaccessible, the agent cannot compare its changes with runtime evidence and humans cannot audit the decision path. That quickly turns automation into review overhead instead of control.

Q: Should organisations separate agent telemetry access from human analyst access?

A: Yes. Agents need structured APIs and query outputs they can reason over, while humans need richer investigative views and audit context. Mixing the two creates overexposure for the agent and clutter for the operator, so the safer pattern is separate interfaces built for different consumers.

Technical breakdown

What the agent harness actually controls

A coding agent is more than a model wrapped in a prompt. The harness is the surrounding system that manages context, tool calls, filesystem access, permission checks, retries, and feedback. It is what turns an LLM from a text generator into a software actor that can execute work in a development environment. The key technical point is that reliability comes from structure: break tasks into steps, persist state, and verify outputs at each stage. Without that scaffolding, the agent loses track of work, over-reasons, or fails to validate what it changed.

Practical implication: Treat the harness as a governed control plane and enforce least privilege for every tool, path, and environment the agent can reach.

Why traces become the source of truth

In traditional software, source code explains the application. In agentic software, code becomes scaffolding and traces capture the real decision path. Traces show which tools the agent invoked, how often it looped, where failures appeared, and whether the runtime behaviour matched the intended task. That makes telemetry a functional dependency, not just an observability add-on. If an agent cannot query traces, it is effectively acting without documentation, which turns debugging and verification into guesswork rather than evidence-based correction.

Practical implication: Make trace access and evaluation data available to the agent, but constrain that access to the specific projects and tasks it is authorised to inspect.

How skills and programmatic observability close the loop

The article describes skills as structured, composable units of method that tell an agent not only what tools exist but when and how to use them. Combined with programmatic observability, skills let the agent test, inspect, and improve its own work using machine-readable telemetry rather than human dashboards. That matters because dashboards are built for people, while agents need APIs, query interfaces, and structured outputs they can reason over during execution. The result is a closed loop in which the agent instruments its task, runs changes, queries evidence, and iterates.

Practical implication: Standardise machine-readable telemetry interfaces and define reusable diagnostic skills so agent behaviour stays auditable and bounded.

NHI Mgmt Group analysis

Coding agents are becoming a governed identity problem, not just a software productivity problem. Once an agent can select tools, retain context, and act across a development workflow, it behaves like a non-human identity that needs explicit access boundaries. That is why this topic belongs in the same governance conversation as workload identity and secrets control. The practical conclusion is that teams should manage agent permissions as runtime authority, not as a loose extension of developer tooling.

Telemetry is now part of the control surface for agentic software. The article is right to frame traces as the source of truth, because agent behaviour is no longer fully inferable from code alone. That creates a new governance layer in which verification depends on the integrity, completeness, and accessibility of telemetry. Runtime observability gap: if the trace layer is incomplete or inaccessible, the organisation cannot prove what the agent actually did. Practitioners should treat that gap as a security and audit failure, not an instrumentation inconvenience.

Self-improvement loops only work when the feedback channels are trustworthy. Agents that can instrument, query, and revise their own work are effectively operating inside an internal control loop. If the loop consumes noisy traces, weak evaluations, or overly broad context, the agent will optimise the wrong thing at machine speed. That is the same pattern identity teams see when lifecycle state and access state diverge. The conclusion is simple: the feedback loop must be as governed as the code path it is trying to improve.

Agentic engineering will increase pressure on identity, secrets, and delegated access controls. Coding agents need credentials, APIs, and tool permissions to operate, which means they inherit many of the failure modes that already affect NHIs and automation accounts. The broader lesson is that agent deployment expands the attack surface unless access is short-lived, traceable, and tied to specific tasks. Security teams should expect agent governance to converge with NHI governance rather than remain a separate practice.

What this signals

The operational signal for practitioners is that agent adoption will outpace control maturity unless telemetry, permissions, and evaluation are designed together. That makes agent governance a programme-level issue, not a tooling choice, because the same access patterns that enable faster development also widen the audit surface for code provenance, change validation, and incident response.

Verification trust gap: when an agent can only be trusted if its traces are trustworthy, the organisation has moved from review-based assurance to control-loop assurance. That will push security and engineering teams to define which telemetry is authoritative, how long it is retained, and who can query it. The teams that standardise those decisions early will have a better path to scaling agentic development safely.

For identity and AI security programmes, this also means workload and agent identity controls need to converge with runtime observability. Access tokens, tool permissions, and trace visibility should be governed as one system because the agent’s authority and its evidence trail are inseparable in practice.

For practitioners

• Define agent tool boundaries Scope each coding agent to the minimum set of repositories, commands, and services required for its task. Record those boundaries as policy so a review can show exactly where the agent was allowed to act.
• Require trace-backed verification Make every agented change produce trace evidence, evaluation results, and a change summary before merge approval. That gives reviewers evidence of runtime behaviour instead of relying on the patch alone.
• Separate human and machine interfaces Expose telemetry to agents through APIs and query interfaces, while reserving dashboards and deeper investigation views for humans. The goal is machine-readable observability without giving the agent unrestricted analyst-grade access.
• Version the harness and skills Treat the agent harness, prompt structure, and reusable skills as controlled assets with versioning and rollback. If an agent behaves unexpectedly, you need to know which control layer changed first.

Key takeaways

• Coding agents change the security problem from code review to runtime verification, because their behaviour emerges through tools, traces, and feedback loops.
• If telemetry is incomplete or inaccessible, self-improving software becomes self-justifying software, which is a control failure as much as an engineering one.
• Agent governance should be built like NHI governance, with scoped access, auditable actions, and evidence-based verification at the centre of the design.

Key terms

• Agent Harness: The agent harness is the runtime layer that wraps a model and turns it into an acting system. It usually includes the loop, tools, context handling, permissions, hooks, and logs. In security terms, it is often the real place where privilege sits and where identity evidence must be governed.
• Runtime telemetry: Observation of what a system actually does while it is executing. In agentic CI/CD, this means seeing which commands, files, tools, and credentials an agent touched so security teams can detect misuse that static workflow review will miss.
• Skill: A skill is a modular instruction package that teaches an agent how to perform a task at runtime. It can include a markdown instruction file, metadata, scripts, and supporting documents. In agentic environments, a skill is not passive documentation. It is an active control input that can shape behaviour, tool use, and execution.
• Trace Verification: Trace verification is the independent checking of what an agent actually did against what it claimed to have done. It matters because agents can report success even when they failed or diverged, so verification must sit outside the agent's control and be auditable in its own right.

What's in the full article

Arize's full article covers the operational detail this post intentionally leaves for the source:

• Examples of how coding-agent harnesses are wired to tracing and evaluation tools in practice
• Discussion of programmatic observability interfaces that let agents query runtime evidence directly
• Implementation details for skills, telemetry access patterns, and verification workflows in agentic software
• The article's specific examples of how teams use trace data to diagnose agent failures and refine code

👉 Arize's full article covers the harness, telemetry, and verification details behind agentic engineering

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and identity lifecycle control. It gives security and identity practitioners a practical base for governing non-human actors as their access patterns become more dynamic.