Join our Newsletter — 33% off our NHI Course
Home› Guides› Observability Platform Security: Controlling Human Query Access to…
Guide SOC & Detection Engineering

Observability Platform Security: Controlling Human Query Access to Production Telemetry

PDF for newsletter subscribers · free to read online

← All guides
By NHI Mgmt Group, edited by Lalit Choda Updated 14 November 2026 13 min read
On this page

An engineer who is refused access to the production database can often read the same customer records, tokens and secrets by searching the logs and traces in the observability platform. Logs, spans and session replays carry request parameters, error payloads and headers, so query access to telemetry is production data access in practice. Most organisations govern it far more loosely than database or console access: every engineer gets estate-wide search, exports are unrestricted, and nobody reviews who searched for what. This guide covers human production access through observability tools: scoping query access by data set and field, break-glass and just-in-time access to restricted telemetry, auditing searches and exports, and reviewing access. It is for platform, SRE and security teams. API keys and ingest tokens are covered in the Datadog API and Application Keys Security Guide and the Elastic API Keys Security Guide, and platform hardening in Securing the monitoring and observability platform itself.

Key takeaways

  • Treat telemetry query rights as production data access. Classify log indexes, trace services and replay data by the most sensitive content they can hold, and grant access with the same approval as the source system.
  • Restrict by data set and field, not only by role. Row and field level controls in observability platforms (restricted datasets, document and field level security, label-based access control) let a broad engineering population keep useful access while sensitive data stays scoped.
  • Know how combined rules resolve. In several platforms the most permissive grant wins: an unrestricted role or a team with no rules can silently cancel the restriction you built.
  • Use break-glass for restricted telemetry. Incident responders sometimes need the raw data; give it through a time-bound, approved and logged elevation rather than standing access.
  • Audit queries and exports. Turn on query audit events where the platform offers them, protect the audit trail, and review exports and wide searches over sensitive data sets.

Why query access is production access

OpenTelemetry instrumentation, framework error handlers and debug logging routinely capture values that never appear on a dashboard: query strings with email addresses, authorisation headers, request bodies, stack traces with connection strings, and replayed user sessions. The Observability, telemetry and SRE for security teams guide covers reducing this at source. Even with good redaction, some sensitive data will land, and the question becomes who can search it.

Telemetry typeSensitive content it commonly holdsDefault exposure risk
Application logsPersonal data in messages, tokens in debug lines, payment referencesFull-text search across all services by any user with log read
APM traces and span attributesSQL statements with parameters, HTTP headers, URLs with query strings, user IDsService-wide trace search; attributes visible in trace views
Real user monitoring and session replayScreen content, typed values if masking is off, user journeysReplay viewable by anyone with product access
Security and audit logsSign-in events, IP addresses, privileged activityMixed with operational data in shared indexes
Infrastructure and process dataCommand lines with arguments, environment variables, host namesCommand-line arguments can include passwords passed as flags

Credential discovery in telemetry maps to ATT&CK T1552 Unsecured Credentials, and bulk reading of a shared repository of operational data resembles T1213 Data from Information Repositories. An attacker with a stolen engineer session (T1078 Valid Accounts) gets whatever that engineer can search, so every unnecessary grant is also an attacker grant.

Designing a data access model for telemetry

  1. Classify data sets: label each index, log pipeline, service and replay project as general, confidential (personal data, customer content) or restricted (secrets risk, health, payment, security logs). Tag at ingest so the tag can drive access rules.
  2. Define audiences: service teams see their own services; on-call SRE sees operational data across services but not restricted fields; security operations sees security logs; a small privacy-trained group sees confidential payloads.
  3. Choose the enforcement layer: separate indexes or tenants for the strongest boundary, row and field level rules within shared stores where separation is too costly.
  4. Default to restricted for new data: untagged data is the usual leak path. Where the platform allows, make data outside any defined set visible only to explicitly granted groups.
  5. Grant through groups from the identity provider: map platform teams and roles to IdP groups so joiners, movers and leavers flow automatically.
ApproachStrengthTrade-offUse when
Separate tenants, projects or clustersStrongest isolation; simple to reason aboutCross-service correlation is harder; more cost and adminRegulated data (payment, health), security logs
Separate indexes or data sets with role grantsClear boundaries inside one platformIndex sprawl; routing must be right at ingestConfidential data with a defined audience
Row level rules (by tag, label or query)Fine-grained; one shared storeRules can be cancelled by broader grants; aggregations may leak countsTeam scoping across many services
Field level rules and maskingKeeps records useful while hiding valuesField names must be known and stableHiding headers, user identifiers, request bodies

Platform controls for scoped query access

Platforms are listed alphabetically. Each section describes the vendor's documented controls as of November 2026; confirm against current documentation, and see each platform's own guide on nhimg.org.

Datadog

  • Data Access Control and Restricted Datasets: admins define a Restricted Dataset by product and tag (up to 10 key:value pairs) and grant it to up to 50 teams, roles or agents; individual users get access through those groups. Covered telemetry includes logs, APM traces, RUM sessions, custom metrics, Error Tracking issues and security signals.
  • Restricted mode: setting a telemetry type to Restricted makes data outside any dataset visible only to users with explicit or unrestricted access. Unrestricted User Groups see all data of a type, so keep them small.
  • Behaviour to plan for: monitors run as a system user with full data access, shared dashboards show data based on the creator's permissions, and anyone who can use a Bits AI agent granted access may see restricted data it surfaces. Logs restriction queries are a separate feature and both apply when used together.
  • Availability: policies are configured under Organization Settings, Access Control, by the Datadog Admin role or the user_access_manage permission; dataset limits differ by plan. See the Datadog API and Application Keys Security Guide.

Elastic

  • Document level security: a role's indices entry takes a query that limits which documents the role can read, with Mustache templates such as {{_user.username}} for per-user rules.
  • Field level security: field_security with grant and except lists controls which fields a role reads; except must be a subset of grant, and multi-fields need both the field and its wildcard listed.
  • Behaviour to plan for: across multiple roles, document queries combine with OR and granted fields as a union, and any role with unrestricted access to an index disables the restriction for that index. Aggregations can still reveal field names and term counts. The documentation suggests separate indices when you need to restrict both documents and fields.
  • Availability: configured through the role API or Kibana custom roles; check your subscription for feature availability. See the Elastic Security Guide.

Grafana

  • Label-based access control (LBAC) for data sources: label selectors are assigned to teams and Grafana adds them to each query, so the backend returns only matching data. Loki and Prometheus are generally available; traces use resource attributes such as resource.service.name.
  • Cloud access policy label selectors: access policies with logs:read or metrics:read scope can carry selectors, for example excluding lines labelled secret="true".
  • Behaviour to plan for: a user in no team with LBAC rules can query all logs or metrics, multiple rules on a team combine so a match on any rule returns data, and administrators in a team with rules are subject to them. Grafana recommends a dedicated LBAC data source separate from the one used by full-access teams.
  • Availability: Grafana Cloud, and Grafana Enterprise with Grafana Enterprise Logs or Metrics; only users with the data source Admin permission edit rules. See the Grafana Security Guide.

Break-glass and just-in-time access to restricted telemetry

During a major incident a responder may need request bodies or a restricted security index. Standing access for that rare case is how restricted data ends up readable by everyone on call. Use elevation instead.

  • Elevation by group: grant the restricted dataset, role or team through an IdP group that your JIT or PAM tool adds members to for a fixed window, for example four hours, tied to an incident ticket. See the JIT Access Guide.
  • Approval proportional to sensitivity: self-approval with ticket reference for confidential operational data; second-person approval for restricted data sets such as payment or health payloads.
  • Break-glass account: keep one documented emergency path for when the IdP or JIT tool is down, with alerts on every use. See the Break-Glass Account Guide and the Break Glass Access Procedure Template.
  • Post-use review: after the window closes, review the queries and exports run under elevation against the incident's purpose.
  • Automatic expiry: NIST SP 800-53 AC-2(2) Automated Temporary and Emergency Account Management describes removing or disabling temporary and emergency accounts after a set period; apply the same rule to elevated group membership.

Exports, sharing and AI assistants

  • Exports and downloads: CSV export and API download move telemetry out of the platform's access model. Limit export rights to roles that need them and log every export.
  • Shared dashboards and notebooks: a dashboard shared outside the team may render with the creator's data rights rather than the viewer's, as Datadog documents for shared dashboards. Review shared links on a schedule.
  • Monitors and scheduled reports: alerts and reports run with system or owner rights and can post restricted data into chat channels. Keep alert payloads to identifiers and links, not raw log lines.
  • AI assistants in the platform: an assistant or agent that queries telemetry on a user's behalf must be scoped like a user; if it has broader access, anyone who can prompt it inherits that access.

Where identity fits

Human query access depends on identity hygiene: SSO with phishing-resistant MFA, group-based grants from the IdP, and leavers removed promptly. Service accounts and application keys created by engineers for dashboards or scripts often carry the creator's full data rights. Treat each one as a non-human identity with an owner and a narrower scope than the human who made it. See How to assign owners to orphaned service accounts.

Implementation approach

  1. Discover (weeks 1 to 2): export current roles, teams and data source permissions; list who can search each index or product; run credential and personal data pattern searches to find the most sensitive data sets.
  2. Classify and tag (weeks 2 to 4): add classification tags or labels at ingest and route restricted content to separate indexes or data sets.
  3. Pilot (weeks 4 to 6): apply row and field rules to one restricted data set and one pilot team, in a dedicated data source or dataset; confirm on-call workflows still work and that dashboards render as expected.
  4. Enable audit (in parallel): turn on query audit events and ship them to a store the platform's own administrators cannot alter.
  5. Roll out: extend to all confidential and restricted data sets; switch untagged data to restricted where supported; set up JIT elevation.
  6. Roll back: keep the previous role mappings exported; if an incident response is blocked, grant time-boxed elevation rather than removing the rule.
  7. Review quarterly: certify membership of unrestricted groups, export rights and elevation history. See the Access Reviews Guide.

Detection and measurement

PlatformDocumented audit capability for query and access activity
DatadogAudit Trail records authentication events, API and application key access and creation, dataset and restriction policy changes, and Archive Search activity; retention is configurable up to 90 days with archiving to cloud storage; Audit Trail Read and Audit Trail Write permissions
Elasticxpack.security.audit.enabled turns on audit logging (off by default); xpack.security.audit.logfile.events.emit_request_body adds request bodies so search queries can be audited, with a documented warning that sensitive data may then appear in audit events; ignore_filters policies reduce noise
GrafanaAuditing in Grafana Enterprise and Cloud logs the query action on data sources, permission changes, logins, API keys and service accounts; log_datasource_query_request_body is off by default because it can expose sensitive data

Query bodies in audit logs are themselves sensitive: protect the audit store (NIST SP 800-53 AU-9) and restrict who can read it.

  • Restricted coverage: percentage of confidential and restricted data sets under row, field or tenant controls.
  • Standing access: number of humans with unrestricted access per telemetry type, trending down.
  • Elevation hygiene: percentage of restricted-data elevations with a ticket and post-use review.
  • Export volume: exports from confidential data sets per month, with each one attributable.
  • Secret findings: credential pattern hits in searchable telemetry per week, and median time to rotate and purge.
  • Review completion: quarterly access review completion for unrestricted groups and export rights.

Common failure patterns

  • One permissive role cancels the model: a legacy "all logs" role or a team without rules gives full access despite new restrictions, because grants combine permissively.
  • Untagged data stays open: new services ship logs without the classification tag and land in the unrestricted default.
  • Monitors leak what users cannot see: alerts running with system rights paste raw restricted log lines into shared channels.
  • Exports without trace: engineers download large result sets for analysis, and nobody can say later what left the platform.
  • Audit logs that leak: request body logging is switched on for query audit, and the audit index is readable by the same broad group.
  • Elevation that never ends: incident access granted by hand is not removed, so the restricted group grows with every incident.

Framework mapping

Control areaNIST SP 800-53 Rev 5NIST CSF 2.0ISO/IEC 27001:2022 Annex A
Scoped query accessAC-3 Access Enforcement; AC-6 Least PrivilegePR.AA-05A.8.3 Information access restriction
Break-glass and JITAC-2(2) Automated Temporary and Emergency Account ManagementPR.AA-05A.8.2 Privileged access rights
Field maskingAC-3PR.DS-01A.8.11 Data masking
Query and export auditAC-6(9) Log Use of Privileged Functions; AU-6 Audit Record Review, Analysis, and ReportingDE.CM-03A.8.15 Logging
Protecting audit dataAU-9 Protection of Audit InformationPR.DS-01A.8.15 Logging
RetentionSI-12 Information Management and RetentionPR.DS-01A.8.10 Information deletion

Practitioner checklist

  • Every log index, trace service and replay project carries a classification tag set at ingest.
  • Confidential and restricted data sets are scoped by tenant, data set, row or field rules.
  • Unrestricted groups per telemetry type are named, small and reviewed quarterly.
  • Untagged data defaults to restricted where the platform supports it.
  • Restricted telemetry is reached through time-bound, ticketed elevation with automatic expiry.
  • A documented break-glass path exists and every use alerts.
  • Query audit is on, shipped to a protected store, and the audit store has its own narrow readership.
  • Export rights are limited and every export is attributable.
  • Monitors, shared dashboards and AI assistants are checked for data rights broader than their audience.

Standards and references

Related NHI Mgmt Group resources: Observability, telemetry and SRE for security teams · Securing the monitoring and observability platform itself · Monitoring and observability platform rollout playbook · Privacy in Security Logging and Monitoring Guide

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Researched and written by the NHI Mgmt Group editorial team using AI tools, under the editorial direction of Lalit Choda. Facts are drawn from the sources listed on this page and checked by our automated quality process. Last updated 14 November 2026. Spotted an error? Use our contact page.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.