Observability, telemetry and SRE for security teams: Using OpenTelemetry pipelines, sampling and SLOs for defence

Most organisations already collect traces, metrics and logs for reliability, yet security teams rarely use that telemetry, rarely secure the pipeline that carries it, and often lose the evidence they need to sampling and cost cuts. This guide shows security practitioners how to work with the observability stack their platform and SRE colleagues run: OpenTelemetry and its OTLP protocol, the Collector as a telemetry pipeline, sampling and cost decisions, service level objectives and burn-rate alerting, and application performance monitoring (APM) traces as investigation data. It is written for detection engineers, SOC leads and platform engineers who want one shared telemetry foundation rather than two parallel ones. Facts about OpenTelemetry come from the specification and project documentation as checked on 30 October 2026; component behaviour changes between releases, so confirm against the version you run.

Key takeaways

• Observability data is security data. Traces show which service called which, with what result and how long it took; that is exactly the path an investigator wants to reconstruct after an API abuse or credential misuse incident.
• The pipeline is an attack surface. OpenTelemetry Collectors hold backend API tokens and TLS keys, listen on ports 4317 and 4318, and can be told to drop or rewrite data, so they need the same hardening as any log forwarder.
• Sampling is a security decision. Head sampling at 1% may be fine for latency dashboards and useless for an investigation; keep errors, authentication and privileged paths with tail sampling, and never sample audit logs.
• SRE methods transfer well. SLOs, error budgets and multiwindow burn-rate alerts give security services (identity providers, detection pipelines, secrets stores) measurable reliability and far fewer noisy pages.
• Redact in the pipeline, not after. Telemetry routinely captures tokens, headers and personal data; the Collector's redaction processor works on an allow-list that fails closed, which is the safe default.

Why observability belongs in the security toolkit

Google's SRE book defines monitoring as collecting, processing and displaying real-time quantitative data about a system, and separates white-box monitoring (internal metrics, logs, profiling) from black-box monitoring (what an external user sees). Security operations have long relied on white-box logs. Observability adds two things logs alone rarely provide: causal links between services, and consistent measurement of normal behaviour.

The point is not to replace the SIEM. Audit logs from identity providers, cloud control planes and operating systems remain the system of record (see the SIEM Guide and the Endpoint Telemetry Guide). Observability telemetry fills the gap between those logs: what the application actually did with the request after authentication succeeded.

OTLP and the OpenTelemetry Collector

OTLP (OpenTelemetry Protocol) is the wire protocol between SDKs, Collectors and backends. The exporter specification, marked Stable, fixes the configuration names and defaults that security teams will see in every deployment manifest.

The OTLP specification also defines partial success responses and retryable versus non-retryable errors (for example, RESOURCE_EXHAUSTED is retryable only when the server returns RetryInfo). This matters for security because a backend under load can silently reject a share of spans or log records. Collector and exporter self-metrics, scraped from port 8888 in the Collector configuration documentation examples, are where those rejections become visible.

Agent and gateway patterns

• Agent: a Collector beside each workload (a sidecar or DaemonSet). It enriches data with host and Kubernetes metadata and buffers locally, but every node then holds exporter credentials.
• Gateway: a central pool of Collectors that receives from agents or SDKs and exports to backends. Backend credentials live in fewer places, and redaction and sampling are enforced once. Tail sampling needs this pattern, because all spans for a trace must reach the same Collector instance.
• Combined: agents forward over mutual TLS to a gateway that holds the only backend tokens. This is usually the best balance for security teams.

Hardening the telemetry pipeline

The OpenTelemetry project publishes Collector security guidance. The practical controls are below.

The project's sensitive data guidance is explicit that OpenTelemetry cannot decide what is sensitive in your context: implementers are responsible for reviewing what instrumentation libraries emit, including authentication credentials and session tokens, and for applying data minimisation. Pair this with the Privacy in Security Logging and Monitoring Guide.

Sampling, retention and cost

The OpenTelemetry sampling documentation calls sampling one of the most effective ways to reduce observability cost, notes that for high-volume systems a rate of 1% or lower can represent the rest accurately, and suggests considering it from around 1,000 traces per second. It also lists when not to sample, including when regulation prohibits dropping data. Representativeness is a statistical property; an investigation needs the specific trace, not a representative one. That is the core tension for security.

A security-aware policy set: keep 100% of traces with error status, 100% of traces touching authentication, token, administrative or payment routes (via attribute or OTTL conditions), 100% of traces from service accounts flagged as privileged, then a probabilistic share of the rest. Put rate limits on the probabilistic tier, not on the security tiers. Audit logs (identity provider, cloud control plane, Kubernetes audit) should bypass trace sampling entirely and follow your log retention policy; see the Cloud Logging and Audit Trail Guide.

Cost levers that do not hurt detection: drop high-cardinality metric labels that nobody queries, aggregate health-check spans, shorten retention for debug-level logs, and route unsampled data to low-cost storage if the budget allows, as the OpenTelemetry guidance suggests. Record every drop rule with an owner, because a filter written to save money is indistinguishable in effect from an attacker's filter.

Applying SRE methods to security services

The SRE book defines an SLI as a carefully defined quantitative measure of some aspect of service, an SLO as a target for that SLI, and an SLA as an agreement with consequences. It recommends treating metrics as distributions and using percentiles rather than averages, and it names four golden signals: latency, traffic, errors and saturation. Each maps cleanly onto security infrastructure.

The example targets are illustrations to adapt, not recommendations from a standard. Freshness and completeness SLOs for the detection pipeline are the most valuable for security: a SIEM that is up but 40 minutes behind fails silently.

Burn-rate alerting

The Google SRE Workbook defines the error budget as the allowed number of bad events, and burn rate as how fast the service consumes that budget relative to the SLO (a burn rate of 1 spends exactly the budget over the window). For a 99.9% SLO it recommends multiwindow, multi-burn-rate alerts:

The Workbook judges alerts on precision, recall, detection time and reset time, which is a useful vocabulary for detection engineering too. The SRE book's paging rules (page only for what is urgent, actionable and user-affecting; automate anything that is purely mechanical) are the same discipline described in the Detection Engineering Guide. A sudden error-budget burn on an authentication service can also be the first sign of credential stuffing or a broken federation trust, so route it to both SRE and the SOC.

APM and traces in investigations

• Correlation keys: W3C Trace Context carries traceparent (version, trace ID, parent ID, trace flags) and tracestate between services. OpenTelemetry logs carry the same TraceId and SpanId, so an application security event can be joined to the full request path.
• Treat inbound context as untrusted: any client can send a forged traceparent. Use it to link records, never to make an authorisation or trust decision, and consider restarting traces at the internet edge.
• What to add: the authenticated principal (user or workload identity), the client ID of the OAuth application, tenant, and authorisation decision outcome as span attributes, subject to redaction and privacy review. Standard HTTP attributes alone do not tell you who made the call.
• Useful hunts: spans where a service calls a data store it has never called before; sudden fan-out from one API key; 403 rates by principal; long spans on export endpoints; new service.name values appearing in production.
• AI and agent workloads: the GenAI conventions extend the same model to model calls and tool execution. See the OpenTelemetry GenAI Semantic Conventions for Security Teams and the LLM Observability Guide.

Where identity fits

Observability platforms are dense with non-human identities: exporter API keys in OTEL_EXPORTER_OTLP_HEADERS, Collector service accounts with cluster-wide read access, ingestion tokens for every agent, and third-party credentials stored for integrations. The Sumo Logic breach began with a compromised credential used against one of the vendor's AWS accounts, and customers were advised to rotate API access keys, installed collector credentials and third-party credentials stored for data collection. Inventory these credentials, scope ingest tokens to write-only, rotate them on a schedule, and log their use. See the Identity Telemetry Requirements for the identity events the pipeline should carry.

Implementation approach

1. Inventory: list every SDK, agent, Collector, backend and token. Record where OTLP endpoints listen and who can reach them.
2. Agree ownership: one shared pipeline, with SRE owning availability and security owning redaction rules, the security sampling tier and access control. Write it down.
3. Harden first: bind receivers, enable TLS and authentication, remove debug extensions, move tokens to a secret store, add memory_limiter.
4. Redaction in audit mode: run the redaction processor with summary output on a copy of production traffic, compare what would be removed, then switch to enforcement for one service before widening.
5. Sampling pilot: introduce tail sampling at the gateway for one service ring, confirm that error and authentication traces are kept at 100%, and compare investigation queries before and after.
6. SLOs for security services: define SLIs for the identity provider, detection pipeline freshness and Collector delivery; add burn-rate alerts.
7. Detections: build hunts on trace data and wire Collector self-metrics into the SOC.
8. Rollback: keep previous Collector configuration versioned and deployable; for sampling, a fallback policy of keep-all at a rate limit avoids losing evidence while you fix a broken rule.

Small versus large organisations: a small team with tens of traces per second should usually not sample at all and can run a single hardened Collector. Large estates need a gateway tier with trace-aware load balancing, per-team ingest tokens and formal change control on pipeline configuration.

Detection and measurement

• Pipeline integrity: alert on Collector refused, dropped and failed-export counts, exporter queue saturation, and sudden falls in span or log volume per service.name.
• Configuration change: log and review every change to Collector configuration, sampling policies and redaction rules. Disabling or modifying security tooling is ATT&CK T1685 Disable or Modify Tools, and a filter that drops audit data has the same effect.
• Credential use: monitor ingest and query tokens for use from unexpected networks, consistent with T1078 Valid Accounts.

Metrics that show it is working:

1. Detection pipeline freshness SLO attainment (target set per tier).
2. Telemetry delivery rate from Collector to backend (exported over received).
3. Share of production services emitting traces with an authenticated principal attribute.
4. Number of secrets or personal data items found in backend telemetry by periodic scanning (target zero).
5. Share of security-relevant traces retained under the sampling policy (target 100%).
6. Alert precision for SLO-based security pages, reviewed monthly.

Common failure patterns

• Uniform head sampling: a 1% ratio sampler set for cost drops 99% of the abuse traces an investigation needs.
• Open receivers and debug ports: Collectors deployed from example configuration with 0.0.0.0 bindings for OTLP, pprof and zpages.
• Tokens everywhere: backend API keys in every agent's environment, never rotated; one leaked key exposes all ingest or, worse, query access, as the Sumo Logic customer advice illustrates.
• Telemetry as a data leak: instrumentation capturing Authorization headers, query strings with tokens or full request bodies, with redaction applied only in the backend UI.
• Silent drops: backends returning partial success or rate limits with nobody watching Collector self-metrics.
• Two pipelines: security and SRE each run separate agents on the same hosts, doubling cost and producing data that cannot be joined.

Framework mapping

Practitioner checklist

• Inventory every OTLP endpoint, Collector and backend token, and who can reach or use each.
• Bind Collector receivers and extensions to specific interfaces and require TLS and authentication.
• Move exporter tokens to a secret store, scope ingest tokens to write-only and rotate them.
• Place memory_limiter first in every pipeline and alert on refused and dropped data.
• Deploy the redaction processor with an allow-list, tested in audit mode first.
• Replace uniform head sampling with tail sampling that keeps errors, authentication and privileged paths at 100%.
• Keep audit logs out of trace sampling and under the retention policy.
• Add authenticated principal and client ID attributes to spans after privacy review.
• Define freshness and delivery SLOs for the detection pipeline with multiwindow burn-rate alerts.
• Put Collector configuration and sampling rules under version control and change review.

Standards and references

• OpenTelemetry: OTLP Specification
• OpenTelemetry: OpenTelemetry Protocol Exporter
• OpenTelemetry: Collector configuration best practices
• OpenTelemetry: Handling sensitive data
• OpenTelemetry: Sampling
• OpenTelemetry: Tail Sampling Processor
• Google SRE: Monitoring Distributed Systems
• Google SRE Workbook: Alerting on SLOs

Related NHI Mgmt Group resources: OpenTelemetry GenAI Semantic Conventions for Security Teams · Endpoint Telemetry Guide · LLM Observability Guide · Detection Engineering Guide · SIEM Guide