AI code generation is making observability a governance issue

At a glance

What this is: This interview argues that AI-generated code and agent-assisted development are making observability a governance issue because code review alone no longer explains production behaviour.

Why it matters: IAM and security teams should care because the same shift that obscures software behaviour also weakens trust in runtime evidence, which matters for human, NHI, and autonomous identity governance.

Context

AI-generated code is changing how teams understand system behaviour. When code changes faster than people can review it, the codebase stops being a dependable source of truth for explaining production outcomes.

The governance gap is that observability now has to carry part of the burden once handled by human code inspection. For identity and access teams, that matters because runtime behaviour increasingly reflects a mix of human developers, AI-assisted change, and production systems whose actions must still be attributable and auditable.

Key questions

Q: How should teams govern AI-generated code when they cannot review every change?

A: Teams should shift from source-only assurance to runtime assurance. That means correlating deployments, traces, logs, and outcome metrics so behaviour can be validated after code is generated and released. The practical goal is not perfect review coverage, but a dependable record of what the system actually did in production.

Q: Why do evals not replace observability for AI-shaped systems?

A: Evals only test anticipated behaviour under controlled conditions, while observability shows what happens when real users, edge cases, and production dependencies appear. The two are complementary. Evals help prove intended behaviour before release, but observability is what exposes drift and unintended runtime outcomes.

Q: What are the signs that code review is no longer enough for production assurance?

A: The main signs are rapid code growth, frequent AI-assisted changes, and repeated production surprises that cannot be explained by reading the code alone. When engineers need telemetry to understand basic behaviour, the system has moved beyond code review as a sufficient assurance method.

Q: How should organisations define observability for non-deterministic systems?

A: They should define observability around business-critical outcomes, not just technical success indicators. If a user-facing or revenue-relevant condition can be harmed without a traditional error, that condition needs to be encoded into monitoring, alerting, and investigation criteria.

Technical breakdown

Why the codebase stops being the source of truth

When AI-generated code is produced faster than engineers can read or reason about it, static code review loses explanatory power. The system may still be correct in a narrow sense, but the team can no longer infer runtime behaviour by inspecting source alone. Observability fills that gap by linking telemetry to what the software actually did, not what reviewers expected it to do. In governance terms, this shifts assurance from code intent to observed execution, which is a meaningful control change for any programme that depends on traceability and accountability.

Practical implication: Treat production telemetry as an evidence source, not just an operations tool, when code review can no longer keep pace with change.

Evals and observability serve different control functions

Evals are structured tests of anticipated behaviour, while observability is the check on what happens in production when real users and edge cases appear. The article draws a line between the two because one cannot replace the other. Evals can validate expected outcomes before release, but only observability can surface drift, surprises, and unintended interactions after deployment. For governance, that means AI-assisted development needs both pre-production verification and runtime verification, with a feedback loop between them.

Practical implication: Use evals to test expected behaviour and observability to catch unexpected behaviour, then feed production findings back into test design.

Subjective quality signals are becoming operational controls

For non-deterministic systems, a simple technical success check is not enough. The article's example shows that quality may need to be defined in business terms, such as whether a user streak is at risk, rather than generic infrastructure metrics. That matters because observability starts to encode policy about what counts as acceptable behaviour. Once quality is tied to business impact, telemetry becomes part of governance, not just diagnostics, because teams must decide which deviations matter enough to act on.

Practical implication: Define observability signals around business-critical outcomes so teams can detect the deviations that actually change risk.

NHI Mgmt Group analysis

AI-generated code makes observability part of governance, not merely operations. When code is created and changed faster than humans can fully inspect it, the codebase no longer functions as the authoritative account of system behaviour. The operational record shifts to telemetry, traces, and runtime evidence. That creates a governance obligation to decide which observed behaviours are acceptable, explainable, and attributable.

Evals are necessary but insufficient because they only cover anticipated behaviour. This article makes clear that the real control boundary is between test conditions and live conditions. Teams that treat evals as a substitute for observability will miss the edge cases that only appear in production, which is where most governance surprises emerge.

Runtime evidence is becoming a cross-domain control surface for human, NHI, and autonomous work. As humans and agents co-author systems, assurance depends on whether the organisation can reconstruct what actually happened after deployment. That is as relevant to identity governance as it is to software governance, because accountability now depends on evidence rather than assumption.

Subjective quality definitions are the new observability frontier. The article's business-impact framing shows that generic uptime metrics are too blunt for AI-shaped systems. Teams now have to define what harmful drift looks like in operational terms, then prove those signals are being watched. The practitioner implication is that governance teams should treat observability criteria as policy decisions, not just engineering preferences.

From our research library:

• 70% of organisations grant AI systems more access than they would give a human employee performing the exact same job, according to the 2026 Infrastructure Identity Survey.
• Systems with least-privileged AI access had a 17% incident rate vs 76% for over-privileged systems. Organisations failing to scope AI access properly are 4.5x more likely to experience a security incident, according to the 2026 Infrastructure Identity Survey.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Runtime observability is becoming the control plane for AI-generated change. As systems are modified faster than engineers can manually verify them, governance depends on whether teams can reconstruct behaviour from production evidence instead of source code alone. That changes the evidentiary model for accountability across human developers and AI-assisted workflows.

Observability and evals now serve different parts of the assurance chain. Evals remain useful for anticipated behaviour, but they do not cover the production edge cases that define real operational risk. Teams should prepare for a split between pre-release verification and runtime verification, with the latter carrying more weight as code generation accelerates.

For practitioners

• Define runtime evidence requirements Specify which traces, logs, and request attributes are required to explain production behaviour when code review cannot fully reconstruct intent.
• Separate eval coverage from observability coverage Map which behaviours are tested before release and which signals must remain visible in production when real users create unanticipated paths.
• Tie quality metrics to business impact Document the user or service outcomes that matter enough to trigger investigation, so observability focuses on meaningful deviation rather than raw system noise.
• Treat agent-assisted changes as audit-relevant Require attribution for changes introduced by human developers and AI-assisted workflows so production incidents can be traced back to the real actor chain.

Key takeaways

• AI-generated code reduces the reliability of source code as the primary explanation for production behaviour.
• Observability is taking on governance duties because runtime evidence is now needed to understand and attribute what software actually did.
• Teams need separate controls for anticipated behaviour and live behaviour, or they will miss the failures that only appear in production.

Key terms

• Observability: Observability is the ability to understand the internal state of a system from the data it produces. In security and operations, that means combining logs, metrics, and traces so teams can explain why something happened, not just confirm that something changed.
• Evals: Evals are structured measurement systems for judging whether an AI tool performs well across many runs. They do not try to prove one exact output is correct. Instead, they use scoring, thresholds, and repeated scenarios to show whether the tool is reliably useful in practice.
• Runtime evidence: Cryptographic proof collected from the environment a workload is using, such as image hashes, cloud-signed metadata, boot measurements or code signatures. It is the material a verifier checks to decide whether an identity should be trusted.
• AI-assisted development: A development workflow where an AI system helps generate, refactor, test, or execute code inside the software delivery process. The security issue is not the assistance itself, but that the tool may operate in trusted contexts that also contain credentials, infrastructure access, and other sensitive assets.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.