Why can harness logs disagree with system telemetry?

Why harness logs and system telemetry can tell different stories

Harness logs and system telemetry are often generated at different layers, with different timing, buffering, and filtering rules. That means one may reflect intent or application-level events while the other reflects only what the operating system, container runtime, or cloud collector actually observed. If the wrapper crashes, buffers never flush, or a collector drops events, the two records can diverge.

Clock skew is another common reason the records look inconsistent. Even when both sources are correct, timestamps may not line up cleanly if the harness, host, and upstream telemetry pipeline do not share the same time source or precision. Redaction can also create a mismatch when the harness suppresses tool details that the lower-level telemetry never saw in the first place.

For practitioners, the key point is that disagreement does not automatically mean one source is wrong. It often means each source is answering a different question about the same execution path, so reconciliation has to start with source attribution, collection timing, and the exact layer each log came from.

Common failure modes that create drift between records

Buffered writes can delay or lose harness entries if the process exits unexpectedly, while system telemetry may already have emitted the underlying process or network event. Wrapper code can also record an attempted action that never completed, whereas telemetry may only show the partial side effect, such as a process start with no corresponding file write. In that case, the gap is about execution state, not necessarily deception.

Filtering is another source of disagreement. A harness may intentionally omit sensitive arguments, tool payloads, or internal retries, while telemetry keeps a narrower but more objective trail of system activity. If the collector is sampling, rate-limiting, or suppressing noisy events, the system view can also look cleaner than the harness view even when the underlying action was real.

In incident work, mismatches are most useful when they reveal what the harness can prove but the telemetry cannot, or vice versa. That distinction helps you decide whether the problem is logging design, collector reliability, or an actual attempt to obscure activity.

How to reconcile harness logs with telemetry without over-trusting either source

The most reliable approach is to compare event identity, sequence, and timing across both streams rather than matching only on message text. Correlate by request ID, session ID, process ID, container ID, tool invocation ID, or another stable key if the platform exposes one. Where those keys are missing, look for consistent transitions such as spawn, connect, write, and exit rather than a perfect one-to-one log line match.

Identity Telemetry Requirements is a useful reference when you need minimum event fields, correlation keys, and retention that make cross-source investigation possible. For platform-specific review, Harness Platform Security helps you think about where delegate tokens, API activity, and audit trails should exist so you can tell a missing record from a missing action.

System telemetry should be treated as the higher-integrity source for actual OS, container, and network behavior, but only within the scope of what it can observe. Harness logs are stronger for intent, orchestration, and application context. The practical task is to align them into one timeline, then explain any gap by collection design before treating it as evidence of tampering.

Risk and Threat Considerations

Disagreement between harness logs and system telemetry becomes risky when it hides unauthorized tool use, incomplete execution, or an attempt to erase traces of a workflow that touched secrets, network destinations, or privileged actions. The danger is not the mismatch itself, but the possibility that one layer is being used to mask activity from another layer that should have captured it.

Failure mechanism: Buffered or redacted harness output, collector loss, or clock drift can break the chain between intent and observed system activity, leaving investigators with partial evidence.

Impact: Teams may miss abuse, misclassify a failed action as successful, or overstate coverage because one log source appears complete while the other contains the missing detail.

Practitioner Guidance

What to verify: Confirm that both sources are collecting the same event class, that timestamps are synchronised, and that the harness flushes before termination. If the harness records tool intent but the telemetry shows no corresponding process, network, or file activity, treat that as a collection and execution question before assuming compromise.

Decision rule: If the harness and telemetry disagree on a security-sensitive action, preserve both records and resolve the gap with source-level evidence, not by choosing the friendlier story. If a control decision depends on the action actually occurring, base it on the most authoritative layer that can independently prove execution.

Practitioner takeaway: The safest interpretation is layered, not binary, because harness logs are often a record of intention while telemetry is a record of observed behaviour.

Related resources from NHI Mgmt Group

• What breaks when an AI system logs outputs but not actions?
• What breaks when AI agent logs and system logs do not align?
• What breaks when mobile telemetry SDKs are hidden inside system apps?
• Why does business email compromise need identity telemetry as well as email logs?