Observability and APM Security Requirements: Buyer's Guide

An observability and APM platform receives a copy of almost everything your applications do, from request headers and SQL statements to user identifiers and, too often, tokens and passwords, and its agents run inside every production process. That makes it one of the most sensitive data stores and one of the most privileged pieces of software in the estate, yet it is usually chosen by engineering teams on dashboards, query speed and price per gigabyte. This vendor-neutral buyer's guide sets out the security requirements to write into the selection: data handling, key management, agent and collector security, access control and vendor assurance, with weighted criteria, vendor questions and a proof of concept. It is for platform and SRE leads, security architects and procurement teams. For using telemetry in defence, see Observability, telemetry and SRE for security teams.

Key takeaways

Assume sensitive data will arrive. The OpenTelemetry project states that telemetry always risks capturing sensitive or personal data and that the implementer is responsible for it; require redaction before data leaves your network and controls once it lands.
Ingest keys are non-human identities in every workload. Require separate write-only ingest keys per environment, separate query and admin keys, expiry, rotation without redeployment and an inventory with last use.
Agents run with your application's privileges. Auto-instrumentation and host agents need signed releases, a verifiable update channel and documented permissions; a compromised monitoring update reaches every host.
Prefer open, standard pipelines. OpenTelemetry Protocol (OTLP) and a collector tier you control let you redact, route and change vendor without re-instrumenting.
Test leakage, not just ingestion. In the PoC, send a fake token, password and email address and prove each is removed or masked end to end.

Define requirements

Agree these points with engineering before demonstrations, so security requirements are scored alongside functionality:

Signals and sources: metrics, traces, logs, profiles, real user monitoring and synthetic checks. Real user monitoring and session replay collect data from customers' browsers and devices and widen the privacy scope considerably.
Data classes: what can appear in attributes, URLs, SQL statements, exception messages and log bodies: personal data, payment data, health data, credentials and session tokens.
Deployment model: SaaS backend, self-hosted backend, or hybrid with an on-premises collector tier. Decide whether raw telemetry may leave your network at all.
Residency and retention: region, retention per signal and per data class, and legal hold needs.
Users: developers, SREs, support staff, security analysts and third parties. Production traces can expose customer data to people who would never get database access.
Automation: will the platform trigger actions (runbooks, auto-remediation, scaling, paging), and with what credentials?
Exit: can data, dashboards, alerts and configuration be exported, and is instrumentation vendor-neutral?

Small organisations using a single SaaS platform should focus on key management, SSO and default redaction. Large or regulated organisations should add a collector tier they operate, customer-managed keys, field-level access control and contractual retention and deletion terms.

Capability areas

Data minimisation and redaction

Before egress: support for processing in a collector you control. The OpenTelemetry Collector offers attributes, filter, redaction and transform processors; the redaction processor deletes attributes that are not on an allow list and masks values matching blocked patterns.
Agent defaults: agents that do not capture request bodies, headers such as Authorization and Cookie, query parameters or SQL literals unless explicitly enabled.
Server-side rules: masking and drop rules applied at ingest, with the ability to delete already-ingested data on request.
Front-end telemetry: input masking by default for real user monitoring and session replay, and consent controls.

Keys and credentials

Distinct key types: write-only ingest keys, read or query keys and administrative API keys, each scoped to an account, environment or team.
Expiry, rotation with overlap so workloads can roll keys without an outage, revocation that takes effect quickly, and a list of keys with creator and last use.
Support for short-lived, federated authentication from collectors and CI pipelines where offered, and for authenticating your own collectors with OIDC or mutual TLS, both of which the OpenTelemetry Collector supports.

Leaked observability keys are common because they sit in every container image and configuration file. See the Datadog API and Application Keys Security Guide and the New Relic credentials guide for what key exposure looks like on two common platforms.

Agents, collectors and the update channel

Signed agent packages and container images, published checksums or provenance, a security advisory feed and an SBOM.
Documented privileges for host and Kubernetes agents (kernel modules, eBPF, privileged containers, cluster RBAC) and the option to run with less.
Collector guidance consistent with OpenTelemetry's: do not run as root, minimise components, bind receivers to specific interfaces rather than 0.0.0.0 (OTLP uses ports 4317 for gRPC and 4318 for HTTP), and keep tokens and private keys out of plain-text configuration.
Control over automatic updates: pinning, staged rollout and the ability to delay. The SolarWinds supply chain compromise showed how a trojanised update to IT monitoring software reached thousands of customers.

Access control inside the platform

SSO with MFA and SCIM, roles separating administration, key management, alert configuration and read access.
Data access controls by team, service, environment or index, so production data from a payments service is not visible to every developer.
Restrictions and logging for live debugging features that can capture variables or run code in production processes.

Audit and integrity

Audit logs for sign-ins, key creation and use, role changes, data deletions, redaction rule changes and exports, exportable to your SIEM.
Alerts when telemetry from a source stops or a pipeline is changed, since silencing monitoring is a defence impairment tactic (ATT&CK T1685 Disable or Modify Tools).
OWASP's Logging Cheat Sheet calls for protection against tampering, unauthorised access, modification and deletion, and for excluding session identifiers, access tokens and passwords from logs; ask how the platform supports both.

Vendor assurance and resilience

Independent assurance covering the regions and products used, tenant separation description, vulnerability disclosure policy and incident notification commitments.
A documented response when the vendor itself is breached. The Sumo Logic breach ended in customers rotating keys, so the ability to rotate every credential quickly is a selection criterion, not an afterthought.

Evaluation criteria

Weights are illustrative; raise data handling where customer or regulated data flows through traces and logs.

Criterion	What good looks like	Evidence to request	Weight
Data minimisation and redaction	Safe agent defaults, collector-side redaction, ingest masking, deletion on request	Default capture settings per agent; PoC leakage test	20%
Key and credential management	Scoped write-only ingest keys, expiry, overlap rotation, inventory with last use	Key model docs; console demo	15%
Agent and update security	Signed releases, SBOM, advisories, least-privilege options, controllable updates	Release signing details; privilege matrix	15%
Access control	SSO, MFA, SCIM, data-level access by team or environment, controlled live debugging	Role model; data access test	15%
Audit and pipeline integrity	Complete audit events, SIEM export, alerts on silent sources and rule changes	Audit log sample; alert configuration	10%
Standards and portability	Native OTLP ingest, works with your own collectors, full export	Supported protocols; export formats	10%
Vendor assurance, residency and retention	Scope-matched reports, region choice, configurable retention, breach notification terms	Reports under NDA; contract terms	15%

Common observability threats and the requirement that addresses each (ATT&CK v20):

Threat	ATT&CK	Requirement
Credentials and tokens read from logs and traces	T1552 Unsecured Credentials	Redaction before egress, masking at ingest, data-level access control
Theft of platform API keys	T1528 Steal Application Access Token	Scoped, expiring, rotatable keys with last-use inventory
Trojanised agent or update	T1195.002 Compromise Software Supply Chain	Signed releases, provenance, pinned and staged updates
Silencing monitoring before an attack	T1685 Disable or Modify Tools	Alerts on stopped sources and pipeline or rule changes

Where identity fits

Every instrumented workload holds a platform credential, and integrations for paging, chat, ticketing and cloud metrics add OAuth clients and cloud roles with read access across accounts. Treat these as non-human identities with owners, scopes and rotation, and record them in your NHI inventory. Ask how cloud metric integrations authenticate; read-only federated roles are preferable to long-lived access keys. See the Monitoring and observability platform integration security guide and the NHI Authentication Guide.

Questions to ask vendors

For each agent, which headers, bodies, query strings and SQL literals are captured by default, and how do we turn capture off globally?
Can we redact in our own collector before data leaves our network, and do you accept native OTLP from it?
Which key types exist, can ingest keys be write-only and scoped per environment, and can they expire and rotate with overlap?
Can we list every key with its creator, scope and last use, and how quickly does revocation take effect?
How are agents and container images signed, where are advisories published, and do you provide an SBOM?
What privileges do host and Kubernetes agents need, and what works with fewer?
Can we restrict data access by service, team or environment, and who on your staff can view our telemetry?
Which audit events exist, how long are they retained at no extra cost, and how do we stream them to our SIEM?
How do we delete specific ingested data, for example a leaked token, and how is deletion confirmed?
If you are breached, what do you notify, how fast, and what will you ask customers to rotate?

Red flags

Agents that capture full request bodies or all headers by default.
A single account-wide API key used for both ingest and administration.
Keys that cannot expire, or rotation that requires redeploying every workload at once.
No way to delete specific ingested records before retention ends.
Unsigned agent packages or forced automatic updates with no pinning.
SSO, audit logs or data access controls only in a premium tier.

Proof of concept

Instrument a test service and inject a fake bearer token in the Authorization header, a fake password in a query string and a test email address in a log line.
Confirm agent defaults drop each value; then route through your own collector with redaction and confirm nothing sensitive arrives at the backend.
Create a write-only ingest key for the test environment, try to query data with it (expect failure), then rotate it with overlap and revoke the old key.
Give a test user access to one service only and confirm they cannot query another service's traces or logs.
Stop the test service's telemetry and change a redaction rule; confirm alerts fire and audit events reach your SIEM.
Verify the signature of the agent package and image you deployed, and pin the version.
Request deletion of one ingested record and confirm it is gone; export a dashboard and data sample in the exit format.

Run the PoC on synthetic data only, and revoke all keys and remove agents afterwards. If redaction or key scoping fails, treat it as a blocker rather than a roadmap item.

After selection: measures and common failure patterns

Track: sensitive-value detections in telemetry per week and time to purge; ingest keys older than your rotation standard; keys with administrative scope used by workloads (target zero); agent versions behind the latest signed release; sources silent beyond their expected interval; and users with access to production data outside their services. The Monitoring and observability platform rollout playbook covers rollout.

Debug settings left on: body capture enabled to fix one incident and never turned off.
One key everywhere: the same key in every image, CI variable and laptop, so rotation becomes an outage.
Everyone sees production: developers can query customer data in traces that they could never read in the database.
No collector tier: applications send straight to the vendor, so redaction depends entirely on agent defaults.
Monitoring not monitored: no alert when telemetry stops. See If your monitoring and observability platform is compromised.

Framework mapping

Framework	Reference	Relevance to selection
NIST CSF 2.0	PR.PS-04, DE.CM-09, GV.SC-06	Log records available for monitoring; monitoring of software and runtime environments; supplier due diligence
NIST SP 800-53 Rev. 5	AU-9, AU-11, SC-8, SC-28, CM-7, AC-6, SA-9, SR-3	Log protection and retention, encryption in transit and at rest, least functionality, least privilege, external services, supply chain
ISO/IEC 27001:2022 Annex A	8.15, 8.16, 5.19, 5.23	Logging, monitoring activities, suppliers, cloud services
NCSC Cloud Security Principles	Principles 1, 2, 3, 13	Data in transit, asset protection, separation between customers, audit information for customers
MITRE ATT&CK v20	T1552, T1528, T1195.002, T1685	Threats in the table above

Standards and references

How NHI Mgmt Group can help

We provide independent requirements, RFP and evaluation support. Browse vendors in our products directory or contact us.

Security Requirements for Selecting an Observability and APM Platform: A Buyer’s Guide to Criteria and Testing