A common mistake is assuming existing spans can be updated freely after the fact. OTEL spans are immutable, so latent evaluation data must be handled through a deliberate design. Teams also need to avoid losing important fields to exporter limits or forcing expensive joins at scale, otherwise the telemetry becomes incomplete or too costly to analyse reliably.
Why the span model trips teams up
OTEL spans are designed to describe an event as it happens, not as a mutable record that can be safely rewritten later. That distinction is where most attempts go wrong. Teams often assume evaluation metrics can be attached to a span after the trace is complete, but the practical constraint is that the span lifecycle, export path, and storage shape have already been decided by the time the data exists.
That matters because the metric is usually not just another tag. Evaluation output may arrive later, may be larger than the original trace payload, and may need a different cardinality profile than the span that carried the request path. If you treat those outputs as interchangeable, you end up forcing the tracing system to do work it was not designed to do.
Two consequences follow. First, late mutation breaks the mental model of trace data and creates uncertainty about what is authoritative. Second, if teams try to preserve everything in span attributes, they can run into exporter limits or sampling behaviour that quietly drops the very context they were trying to preserve.
When teams want the evaluation result to stay queryable alongside the trace, the safer pattern is to design for correlation, not post-hoc mutation. That usually means deciding up front which values belong on the span, which belong in a separate metric or event stream, and which identifiers are needed to join them later.
Where scale and fidelity fail in practice
The biggest operational mistake is to assume a join will be cheap just because both datasets carry the same trace or span identifiers. At low volume that may work; at scale it often becomes an expensive cross-system lookup, especially when evaluation outputs are high volume, delayed, or produced asynchronously. The result is a telemetry design that looks complete in testing but becomes slow or brittle in production.
Another common failure mode is overloading spans with fields that are useful for analysis but not safe for every exporter or backend. Span attributes are bounded, and different collectors or backends may cap size, truncate values, or reject noisy dimensions. If the evaluation payload depends on those fields, analysts may later see incomplete traces and misread the absence as a model or system issue rather than a telemetry design issue.
For teams that are also dealing with machine-generated activity or service-to-service requests, the issue can be harder to spot because the telemetry volume is high and the correlation logic feels automatic. In practice, however, the design still needs clear ownership of identity-bearing fields, trace context, and evaluation outputs so that the data can be queried without depending on accidental exporter behaviour. The Ultimate Guide to NHIs is useful background when those traces are produced by services, workloads, or other non-human actors that need consistent governance across systems.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST CSF 2.0 and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | PR.AC-1 — Identities and Credentials Issuance and Management | Correlating evaluation output safely depends on controlled trace identity and access context. |
| PR.DS-1 — Data-at-Rest Protection | Evaluation metrics and trace payloads can carry sensitive operational data that needs bounded handling. | |
| Recommendation — Limit who can publish or alter telemetry identifiers and correlation data. Protect stored telemetry payloads so exported span data remains intact and confidential. | ||
| CIS Controls v8 | 8.6 — Audit Log Storage, Retention, and Protection | OTEL spans and derived evaluation records are log-like telemetry that need retention and integrity controls. |
| 13.4 — Conduct Full Port Security Scans | Telemetry transport and collector endpoints need controlled exposure when spans are forwarded across systems. | |
| Recommendation — Store telemetry in a protected pipeline that preserves integrity and retention requirements. Validate telemetry collection endpoints and forwarding paths before relying on them in production. | ||
Practitioner Guidance
What to verify: Verify whether the evaluation metric is truly a span attribute, a span event, or a separate metric record. If it arrives after the span is closed, do not design as though the span can be edited in place.
Implementation sequence:
- Decide the correlation key before instrumenting, usually a stable trace or span identifier.
- Keep only the minimum fields needed for trace-level context on the span.
- Send evaluation output to a structure that matches its volume and query pattern.
- Test exporter limits and backend truncation with realistic payload sizes.
- Validate the join path under production-like cardinality, not just in a small sample.
Common mistake: The tempting shortcut is to keep adding context to the span until the trace “has everything.” That usually makes analysis harder, not easier, because the tracing layer starts carrying data that belongs in a different telemetry shape.
Practitioner takeaway: Treat the span as the correlation anchor, not the storage container for every evaluation result; the design succeeds when trace context stays lightweight and the metric can survive at production scale.
Related resources from NHI Mgmt Group
- What do security teams get wrong when they try to measure human risk with vanity metrics?
- What do teams get wrong when they try to automate threat modeling too early?
- What do teams get wrong about model evaluation metrics?
- What do teams get wrong when they try to test agent memory with simple replay?
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on September 19, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
