An engineer who is refused access to the production database can often read the same customer records, tokens and secrets by searching the logs and traces in the observability platform. Logs, spans and session replays carry request parameters, error payloads and headers, so query access to telemetry is production data access in practice. Most organisations govern it far more loosely than database or console access: every engineer gets estate-wide search, exports are unrestricted, and nobody reviews who searched for what. This guide covers human production access through observability tools: scoping query access by data set and field, break-glass and just-in-time access to restricted telemetry, auditing searches and exports, and reviewing access. It is for platform, SRE and security teams. API keys and ingest tokens are covered in the Datadog API and Application Keys Security Guide and the Elastic API Keys Security Guide, and platform hardening in Securing the monitoring and observability platform itself.
Key takeaways
- Treat telemetry query rights as production data access. Classify log indexes, trace services and replay data by the most sensitive content they can hold, and grant access with the same approval as the source system.
- Restrict by data set and field, not only by role. Row and field level controls in observability platforms (restricted datasets, document and field level security, label-based access control) let a broad engineering population keep useful access while sensitive data stays scoped.
- Know how combined rules resolve. In several platforms the most permissive grant wins: an unrestricted role or a team with no rules can silently cancel the restriction you built.
- Use break-glass for restricted telemetry. Incident responders sometimes need the raw data; give it through a time-bound, approved and logged elevation rather than standing access.
- Audit queries and exports. Turn on query audit events where the platform offers them, protect the audit trail, and review exports and wide searches over sensitive data sets.
Why query access is production access
OpenTelemetry instrumentation, framework error handlers and debug logging routinely capture values that never appear on a dashboard: query strings with email addresses, authorisation headers, request bodies, stack traces with connection strings, and replayed user sessions. The Observability, telemetry and SRE for security teams guide covers reducing this at source. Even with good redaction, some sensitive data will land, and the question becomes who can search it.
| Telemetry type | Sensitive content it commonly holds | Default exposure risk |
|---|---|---|
| Application logs | Personal data in messages, tokens in debug lines, payment references | Full-text search across all services by any user with log read |
| APM traces and span attributes | SQL statements with parameters, HTTP headers, URLs with query strings, user IDs | Service-wide trace search; attributes visible in trace views |
| Real user monitoring and session replay | Screen content, typed values if masking is off, user journeys | Replay viewable by anyone with product access |
| Security and audit logs | Sign-in events, IP addresses, privileged activity | Mixed with operational data in shared indexes |
| Infrastructure and process data | Command lines with arguments, environment variables, host names | Command-line arguments can include passwords passed as flags |
Credential discovery in telemetry maps to ATT&CK T1552 Unsecured Credentials, and bulk reading of a shared repository of operational data resembles T1213 Data from Information Repositories. An attacker with a stolen engineer session (T1078 Valid Accounts) gets whatever that engineer can search, so every unnecessary grant is also an attacker grant.
Designing a data access model for telemetry
- Classify data sets: label each index, log pipeline, service and replay project as general, confidential (personal data, customer content) or restricted (secrets risk, health, payment, security logs). Tag at ingest so the tag can drive access rules.
- Define audiences: service teams see their own services; on-call SRE sees operational data across services but not restricted fields; security operations sees security logs; a small privacy-trained group sees confidential payloads.
- Choose the enforcement layer: separate indexes or tenants for the strongest boundary, row and field level rules within shared stores where separation is too costly.
- Default to restricted for new data: untagged data is the usual leak path. Where the platform allows, make data outside any defined set visible only to explicitly granted groups.
- Grant through groups from the identity provider: map platform teams and roles to IdP groups so joiners, movers and leavers flow automatically.
| Approach | Strength | Trade-off | Use when |
|---|---|---|---|
| Separate tenants, projects or clusters | Strongest isolation; simple to reason about | Cross-service correlation is harder; more cost and admin | Regulated data (payment, health), security logs |
| Separate indexes or data sets with role grants | Clear boundaries inside one platform | Index sprawl; routing must be right at ingest | Confidential data with a defined audience |
| Row level rules (by tag, label or query) | Fine-grained; one shared store | Rules can be cancelled by broader grants; aggregations may leak counts | Team scoping across many services |
| Field level rules and masking | Keeps records useful while hiding values | Field names must be known and stable | Hiding headers, user identifiers, request bodies |
Platform controls for scoped query access
Platforms are listed alphabetically. Each section describes the vendor's documented controls as of November 2026; confirm against current documentation, and see each platform's own guide on nhimg.org.
Datadog
- Data Access Control and Restricted Datasets: admins define a Restricted Dataset by product and tag (up to 10 key:value pairs) and grant it to up to 50 teams, roles or agents; individual users get access through those groups. Covered telemetry includes logs, APM traces, RUM sessions, custom metrics, Error Tracking issues and security signals.
- Restricted mode: setting a telemetry type to Restricted makes data outside any dataset visible only to users with explicit or unrestricted access. Unrestricted User Groups see all data of a type, so keep them small.
- Behaviour to plan for: monitors run as a system user with full data access, shared dashboards show data based on the creator's permissions, and anyone who can use a Bits AI agent granted access may see restricted data it surfaces. Logs restriction queries are a separate feature and both apply when used together.
- Availability: policies are configured under Organization Settings, Access Control, by the Datadog Admin role or the
user_access_managepermission; dataset limits differ by plan. See the Datadog API and Application Keys Security Guide.
Elastic
- Document level security: a role's
indicesentry takes aquerythat limits which documents the role can read, with Mustache templates such as{{_user.username}}for per-user rules. - Field level security:
field_securitywithgrantandexceptlists controls which fields a role reads;exceptmust be a subset ofgrant, and multi-fields need both the field and its wildcard listed. - Behaviour to plan for: across multiple roles, document queries combine with OR and granted fields as a union, and any role with unrestricted access to an index disables the restriction for that index. Aggregations can still reveal field names and term counts. The documentation suggests separate indices when you need to restrict both documents and fields.
- Availability: configured through the role API or Kibana custom roles; check your subscription for feature availability. See the Elastic Security Guide.
Grafana
- Label-based access control (LBAC) for data sources: label selectors are assigned to teams and Grafana adds them to each query, so the backend returns only matching data. Loki and Prometheus are generally available; traces use resource attributes such as
resource.service.name. - Cloud access policy label selectors: access policies with
logs:readormetrics:readscope can carry selectors, for example excluding lines labelledsecret="true". - Behaviour to plan for: a user in no team with LBAC rules can query all logs or metrics, multiple rules on a team combine so a match on any rule returns data, and administrators in a team with rules are subject to them. Grafana recommends a dedicated LBAC data source separate from the one used by full-access teams.
- Availability: Grafana Cloud, and Grafana Enterprise with Grafana Enterprise Logs or Metrics; only users with the data source
Adminpermission edit rules. See the Grafana Security Guide.
Break-glass and just-in-time access to restricted telemetry
During a major incident a responder may need request bodies or a restricted security index. Standing access for that rare case is how restricted data ends up readable by everyone on call. Use elevation instead.
- Elevation by group: grant the restricted dataset, role or team through an IdP group that your JIT or PAM tool adds members to for a fixed window, for example four hours, tied to an incident ticket. See the JIT Access Guide.
- Approval proportional to sensitivity: self-approval with ticket reference for confidential operational data; second-person approval for restricted data sets such as payment or health payloads.
- Break-glass account: keep one documented emergency path for when the IdP or JIT tool is down, with alerts on every use. See the Break-Glass Account Guide and the Break Glass Access Procedure Template.
- Post-use review: after the window closes, review the queries and exports run under elevation against the incident's purpose.
- Automatic expiry: NIST SP 800-53 AC-2(2) Automated Temporary and Emergency Account Management describes removing or disabling temporary and emergency accounts after a set period; apply the same rule to elevated group membership.
Exports, sharing and AI assistants
- Exports and downloads: CSV export and API download move telemetry out of the platform's access model. Limit export rights to roles that need them and log every export.
- Shared dashboards and notebooks: a dashboard shared outside the team may render with the creator's data rights rather than the viewer's, as Datadog documents for shared dashboards. Review shared links on a schedule.
- Monitors and scheduled reports: alerts and reports run with system or owner rights and can post restricted data into chat channels. Keep alert payloads to identifiers and links, not raw log lines.
- AI assistants in the platform: an assistant or agent that queries telemetry on a user's behalf must be scoped like a user; if it has broader access, anyone who can prompt it inherits that access.
Where identity fits
Human query access depends on identity hygiene: SSO with phishing-resistant MFA, group-based grants from the IdP, and leavers removed promptly. Service accounts and application keys created by engineers for dashboards or scripts often carry the creator's full data rights. Treat each one as a non-human identity with an owner and a narrower scope than the human who made it. See How to assign owners to orphaned service accounts.
Implementation approach
- Discover (weeks 1 to 2): export current roles, teams and data source permissions; list who can search each index or product; run credential and personal data pattern searches to find the most sensitive data sets.
- Classify and tag (weeks 2 to 4): add classification tags or labels at ingest and route restricted content to separate indexes or data sets.
- Pilot (weeks 4 to 6): apply row and field rules to one restricted data set and one pilot team, in a dedicated data source or dataset; confirm on-call workflows still work and that dashboards render as expected.
- Enable audit (in parallel): turn on query audit events and ship them to a store the platform's own administrators cannot alter.
- Roll out: extend to all confidential and restricted data sets; switch untagged data to restricted where supported; set up JIT elevation.
- Roll back: keep the previous role mappings exported; if an incident response is blocked, grant time-boxed elevation rather than removing the rule.
- Review quarterly: certify membership of unrestricted groups, export rights and elevation history. See the Access Reviews Guide.
Detection and measurement
| Platform | Documented audit capability for query and access activity |
|---|---|
| Datadog | Audit Trail records authentication events, API and application key access and creation, dataset and restriction policy changes, and Archive Search activity; retention is configurable up to 90 days with archiving to cloud storage; Audit Trail Read and Audit Trail Write permissions |
| Elastic | xpack.security.audit.enabled turns on audit logging (off by default); xpack.security.audit.logfile.events.emit_request_body adds request bodies so search queries can be audited, with a documented warning that sensitive data may then appear in audit events; ignore_filters policies reduce noise |
| Grafana | Auditing in Grafana Enterprise and Cloud logs the query action on data sources, permission changes, logins, API keys and service accounts; log_datasource_query_request_body is off by default because it can expose sensitive data |
Query bodies in audit logs are themselves sensitive: protect the audit store (NIST SP 800-53 AU-9) and restrict who can read it.
- Restricted coverage: percentage of confidential and restricted data sets under row, field or tenant controls.
- Standing access: number of humans with unrestricted access per telemetry type, trending down.
- Elevation hygiene: percentage of restricted-data elevations with a ticket and post-use review.
- Export volume: exports from confidential data sets per month, with each one attributable.
- Secret findings: credential pattern hits in searchable telemetry per week, and median time to rotate and purge.
- Review completion: quarterly access review completion for unrestricted groups and export rights.
Common failure patterns
- One permissive role cancels the model: a legacy "all logs" role or a team without rules gives full access despite new restrictions, because grants combine permissively.
- Untagged data stays open: new services ship logs without the classification tag and land in the unrestricted default.
- Monitors leak what users cannot see: alerts running with system rights paste raw restricted log lines into shared channels.
- Exports without trace: engineers download large result sets for analysis, and nobody can say later what left the platform.
- Audit logs that leak: request body logging is switched on for query audit, and the audit index is readable by the same broad group.
- Elevation that never ends: incident access granted by hand is not removed, so the restricted group grows with every incident.
Framework mapping
| Control area | NIST SP 800-53 Rev 5 | NIST CSF 2.0 | ISO/IEC 27001:2022 Annex A |
|---|---|---|---|
| Scoped query access | AC-3 Access Enforcement; AC-6 Least Privilege | PR.AA-05 | A.8.3 Information access restriction |
| Break-glass and JIT | AC-2(2) Automated Temporary and Emergency Account Management | PR.AA-05 | A.8.2 Privileged access rights |
| Field masking | AC-3 | PR.DS-01 | A.8.11 Data masking |
| Query and export audit | AC-6(9) Log Use of Privileged Functions; AU-6 Audit Record Review, Analysis, and Reporting | DE.CM-03 | A.8.15 Logging |
| Protecting audit data | AU-9 Protection of Audit Information | PR.DS-01 | A.8.15 Logging |
| Retention | SI-12 Information Management and Retention | PR.DS-01 | A.8.10 Information deletion |
Practitioner checklist
- Every log index, trace service and replay project carries a classification tag set at ingest.
- Confidential and restricted data sets are scoped by tenant, data set, row or field rules.
- Unrestricted groups per telemetry type are named, small and reviewed quarterly.
- Untagged data defaults to restricted where the platform supports it.
- Restricted telemetry is reached through time-bound, ticketed elevation with automatic expiry.
- A documented break-glass path exists and every use alerts.
- Query audit is on, shipped to a protected store, and the audit store has its own narrow readership.
- Export rights are limited and every export is attributable.
- Monitors, shared dashboards and AI assistants are checked for data rights broader than their audience.
Standards and references
- NIST: SP 800-53 Rev. 5 Security and Privacy Controls
- MITRE ATT&CK: T1552 Unsecured Credentials
- Datadog: Data Access Control
- Datadog: Audit Trail events
- Elastic: Controlling access at the document and field level
- Elastic: Auditing settings
- Grafana: Label based access control for data sources
- Grafana: Audit Grafana
Related NHI Mgmt Group resources: Observability, telemetry and SRE for security teams · Securing the monitoring and observability platform itself · Monitoring and observability platform rollout playbook · Privacy in Security Logging and Monitoring Guide