Coding agents need observability contracts, not just auto-instrumentation

At a glance

What this is: This is an analysis of how coding agents are changing observability, with the key finding that auto-generated code still needs explicit telemetry contracts to be supportable in production.

Why it matters: It matters because IAM-adjacent and broader security teams increasingly depend on reliable service telemetry for detection, response, and governance, and agent-written code can erode those signals if instrumentation is inconsistent or misleading.

👉 Read Sawmills' instrumentation guide for observability in the age of coding agents

Context

Coding agents are changing software delivery faster than most teams are updating their observability practices. The core gap is not whether code can be generated, but whether the resulting services can still explain themselves through consistent logs, metrics, traces, and alerts.

In practice, this creates a governance problem for engineering and security teams alike. If agents write more code with less human review, then the organisation needs stronger standards for telemetry, data handling, and service accountability so incidents remain diagnosable and control evidence remains trustworthy.

Key questions

Q: How should teams make coding agents instrument software safely?

A: Teams should give coding agents a machine-readable observability contract before code is generated. That contract should define approved libraries, required span fields, allowed metric labels, severity meanings, and forbidden data such as PII or secrets. Without those constraints, agents will infer patterns from inconsistent repositories and reproduce telemetry debt at scale.

Q: Why do high-cardinality metrics cause so many observability problems?

A: High-cardinality metrics create too many unique time series, which raises cost, slows queries, and can make dashboards unusable during incidents. They are especially risky when agents or hurried developers use request-specific values as labels. Metrics should aggregate behaviour, while traces and logs carry request-level context under tighter control.

Q: What do security teams get wrong about structured logging and traces?

A: Teams often assume that having logs and traces means the service is observable, but observability depends on correlation and meaning. If logs lack trace IDs, metrics lack matching attributes, or event names are inconsistent, responders still cannot reconstruct the failure path. The signals must connect into one narrative.

Q: How can organisations tell whether agent-written instrumentation is actually working?

A: Look for evidence that alerts lead cleanly to dashboards, example traces, log events, and runbooks without manual guesswork. If responders still have to reverse-engineer what a metric means or which log line matters, the instrumentation is not working. The test is whether the system explains a failure in operational language.

Technical breakdown

Why auto-instrumentation is not enough for agent-written code

Auto-instrumentation and eBPF can capture network calls, protocol timing, and many runtime events without code changes, but they cannot infer business meaning. A checkout failure might be a card decline, a provider timeout, or a fraud decision, and those are operationally different outcomes. Language-level instrumentation adds custom spans, bounded attributes, and business events that preserve context. Coding agents tend to optimise for apparent completeness, which can produce noisy telemetry that looks rich but is hard to query or trust. Practical implication: define when automatic telemetry is sufficient and when human-authored spans, labels, and event names are mandatory.

Practical implication: require explicit application-level telemetry for business-critical workflows, not just agent-generated default instrumentation.

How bad metric labels and log severity break observability

Observability fails when telemetry is high-cardinality, inconsistently named, or semantically wrong. Metric labels like user_id or order_id can explode time-series counts, while treating every exception as error creates alert fatigue. Good telemetry uses bounded dimensions, stable event names, and a shared severity model so operators can distinguish expected business outcomes from genuine failures. Coding agents will mirror whatever patterns they see unless the repository states otherwise. That means the control problem is not only what to collect, but what to forbid. Practical implication: publish a service-level schema for labels, severities, and allowed fields before asking an agent to instrument code.

Practical implication: constrain labels, event names, and severity values through policy, review, and tests so agents cannot invent unsafe telemetry.

Why correlation and SLO alignment matter more than telemetry volume

A dashboard full of metrics does not help if alerts, traces, and logs cannot be joined into one operational path. Correlation requires trace IDs, matching attributes, consistent naming, and runbooks that point responders to the right signals. SLO alignment matters because system health is not the same as user success, especially in workflows like checkout, authentication, or payments. Coding agents can generate local instrumentation, but they cannot reliably infer which signals the business uses to define success. Practical implication: align every new telemetry decision to a specific user journey, SLO, dashboard, and runbook before merging code.

Practical implication: tie instrumentation to user journeys and SLOs so operational signals measure business outcomes, not just technical activity.

NHI Mgmt Group analysis

Agent-written code turns observability into a governance problem, not just an engineering task. When a coding agent can edit multiple files, run tests, and prepare pull requests, the question is no longer whether instrumentation exists somewhere in the stack. The question is whether the organisation has defined enough policy for the agent to write telemetry that remains coherent under incident pressure. That makes telemetry standards part of software governance, not an afterthought. Practitioners should treat observability contracts as enforceable control documents.

High-cardinality telemetry is a control failure, not a tuning issue. The article correctly shows that rich-looking metrics can become unusable when developers or agents add unbounded labels and noisy event fields. This is a named failure mode: telemetry cardinality sprawl. Once that happens, dashboards become slow, alerting degrades, and responders lose confidence in the data. The lesson for teams is to govern telemetry fields the same way they govern secrets, because both can quietly undermine operational control.

Instrumentation now depends on company context that agents can read and obey. Most repositories still lack a machine-readable description of approved log schemas, severity meanings, span names, and forbidden fields. That absence forces agents to infer patterns, which reproduces inconsistency at scale. A useful observability contract should state the approved libraries, required attributes, PII restrictions, and dashboard ownership. Practitioners should make those rules discoverable in-repo before allowing code generation.

Security teams should care because observability is also an evidence layer. The same telemetry that helps developers debug services also supports detection engineering, incident scoping, and control validation. If agent-written code leaks raw payloads, secrets, or misleading business events into logs, it weakens both resilience and governance. That means telemetry review belongs alongside secure coding review, especially in services that handle authentication, payments, identity flows, or sensitive data. Practitioners should align observability policy with security and privacy controls.

Observability debt will grow unless teams define the definition of done for agents. The article points to a broader pattern: AI-assisted delivery can accelerate code production faster than organisations can mature their operational standards. That creates observability debt, where functionality improves but supportability lags. Teams should respond by making instrumentation planning, telemetry tests, and runbook updates mandatory pre-merge steps. Practitioners should measure success by whether the system tells the truth when it fails, not by how much telemetry it emits.

What this signals

Telemetry is becoming an identity-adjacent control surface. In environments where coding agents generate more of the application, the evidence trail they leave behind becomes part of governance. That matters for authentication flows, service accounts, API keys, and any workflow where logs and traces can expose or protect sensitive state. Teams should treat observability policy as part of security design, not just platform engineering.

Observability debt will increasingly mirror identity debt. When services accumulate inconsistent labels, undocumented severity rules, and missing correlation, the organisation loses the ability to answer who did what, when, and under what context. In the same way that unmanaged credentials create hidden access paths, unmanaged telemetry creates hidden operational blind spots. Practitioners should expect stronger demand for schema discipline, data minimisation, and reviewable contracts.

Telemetry should now be governed as a product of the software supply chain. Coding agents amplify whichever standards the organisation already has, so weak observability conventions will spread quickly across services. The practical response is to make telemetry tests, runbook linkage, and forbidden-field checks part of CI. That is the difference between instrumentation that records activity and instrumentation that supports accountability.

For practitioners

• Publish an observability contract for every service Define approved logger, tracing library, metric naming, required fields, forbidden fields, and severity meanings in a machine-readable repo file that coding agents must consume before generating code.
• Block high-cardinality fields at review time Reject user_id, order_id, raw path, exception message, and similar unbounded labels in metrics, and route request-level detail to traces or controlled structured logs instead.
• Tie telemetry to business journeys and SLOs Require every new span, metric, and log event to map to a named user journey, an SLO, and a runbook entry before merge.
• Test instrumentation like application logic Add CI checks for log schema compliance, forbidden fields, trace context propagation, allowed label sets, and telemetry coverage for success, handled failure, and unexpected failure.

Key takeaways

• Coding agents do not remove the need for observability, they raise the standard for how telemetry must be specified and governed.
• The biggest failure mode is not missing data but misleading, high-cardinality, or uncorrelated telemetry that looks useful and behaves like noise.
• Teams should treat instrumentation as a contractual part of software delivery, with clear schemas, tests, and runbooks before code reaches production.

Key terms

• Observability contract: A documented set of rules that defines how a service should emit logs, metrics, and traces. It specifies approved libraries, field names, severity meanings, forbidden data, and the signals required to support debugging, incident response, and operational accountability.
• High-cardinality Data: High-cardinality data contains many unique combinations of tags or labels, which makes querying and storage more expensive. In observability, this can reduce the usefulness of telemetry. For identity teams, it also makes access evidence harder to search and correlate at incident speed.
• Telemetry correlation: The process of joining separate security and application signals into one timeline so analysts can interpret them together. For identity work, this means linking sign-in risk with downstream SaaS activity to decide whether an event is suspicious, confirmed, or benign.
• Observability debt: Observability debt is the accumulated inability to explain system behaviour because logging, tracing, and evidence capture were left too shallow for the risks in play. In MCP and agentic systems, it shows up when investigators can see that a tool was used but not what happened next.

What's in the full article

Sawmills' full article covers the operational detail this post intentionally leaves for the source:

• Concrete examples of how to rewrite weak telemetry into stable, searchable event names and bounded metric labels.
• Practical guidance on deciding what belongs in logs, what belongs in traces, and what should never be emitted.
• A service-level observability contract pattern that teams can adapt for CI review and agent instructions.
• The article's detailed breakdown of how to align instrumentation with real SLOs and user journeys.

👉 Sawmills' full guide includes concrete telemetry patterns, severity rules, and contract examples for agent-written code.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management in a way that supports security and platform teams. It helps practitioners translate identity control thinking into the operational standards modern programmes need.