How should teams secure Prometheus queries when metrics must be filtered by tenant or permission system?

Teams should place an authorization layer in front of Prometheus so every query is checked before results are returned. A request proxy can validate the caller’s bearer token, map request parameters to the target object, and enforce a required label in the query. That pattern keeps shared observability data scoped to the right permissions system while still allowing normal Prometheus access patterns.

Why fronting Prometheus with authorization is the right pattern

Prometheus is excellent at collecting and serving metrics, but it is not a tenant-aware authorization engine by itself. If teams need to filter query results by tenant or another permission system, the security boundary has to sit in front of Prometheus, where each request can be authenticated, authorized, and constrained before data is returned. That keeps the observability plane usable without turning raw metrics into an open data source.

The practical design choice is to treat the query path like any other protected API. A proxy can inspect the bearer token, identify the caller, translate request context into an object or tenant scope, and then enforce a label rule or equivalent filter before forwarding the query. For teams dealing with shared telemetry, that is the difference between safe multi-consumer access and accidental cross-tenant disclosure.

A useful implementation detail is to keep the authorization decision close to the request rather than trying to infer safety after Prometheus has already executed the query. Once a query can read across all series, the damage is done even if the caller later only sees a subset. OWASP API Security Top 10 is a helpful lens here because the failure mode is essentially broken authorization on a data API.

What teams need to control in the query path

The core control objective is not just “block bad users”, it is “make every query resolve to the correct permission scope.” That means the proxy or gateway must understand three things consistently: who is calling, what tenant or object the request targets, and what label or filter is required for the result set. If any of those are left to client convention, enforcement becomes optional.

Teams also need to decide whether the policy is deny-by-default or allow-by-default. For shared Prometheus deployments, deny-by-default is the safer posture because a missing label, a malformed selector, or a query rewrite bug should fail closed rather than leak broader telemetry. When the target is a permission system, the authorization model should be explicit enough that query rewriting, UI filtering, and direct API access all produce the same scope.

For practitioners already running shared infrastructure, this pattern is closely aligned with workload and service credential governance. Ultimate Guide to NHIs, Key Challenges and Risks is relevant because scoped access, excessive privilege, and visibility gaps are the same control failures that make metric backends unsafe when they are not wrapped with enforcement.

Why query filtering fails without strong enforcement

Tenant filtering in observability usually fails in one of two ways. The first is trust in the client, where the caller is expected to add the right tenant label and never omit or alter it. The second is partial enforcement, where only the UI or dashboard layer is filtered but direct API calls can still query the full dataset. Both create a hidden path to over-broad access.

There is also a control-plane and data-plane mismatch risk. A system may correctly authenticate the caller but still allow broad metric retrieval if the query engine is not bound to the same authorization context. That is why request parameters, labels, and query templates should be checked together, not separately. The enforcement point should be able to prove that the final executed query matches the intended tenant or permission scope.

Where query access is backed by long-lived credentials or overly broad tokens, the issue becomes an identity exposure problem as well as a query authorization problem. The common pattern is not a broken metric backend alone, it is a privileged access path that was never reduced to the minimum query scope needed for the user or service.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

OWASP Non-Human Identity Top 10 and OWASP Agentic AI Top 10 address the attack and risk surface, while CIS Controls v8 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.

Framework	Control / Reference	Relevance
OWASP Non-Human Identity Top 10	NHI-06 — Overprivileged Access	Scoped metric access depends on least privilege for callers and query credentials.
	NHI-01 — Secrets and Credential Management	Bearer tokens and query credentials are the control point for authorizing Prometheus access.
	NHI-04 — Visibility and Discovery	Shared metrics require clear discovery of who can query which tenant-scoped data.
Recommendation — Enforce least-privilege query access and remove any credential that can read beyond its tenant scope. Protect, rotate, and narrowly scope the tokens used to reach the Prometheus query path. Inventory every query path and verify that each one enforces tenant scoping before data leaves Prometheus.
OWASP Agentic AI Top 10	A4 — Access Control and Permissions	Query execution must be constrained by an explicit authorization decision before any data is returned.
Recommendation — Bind every query to a checked permission context and deny execution when scope cannot be proven.
CIS Controls v8	6 — Access Control Management	The problem is unauthorized access to shared observability data and query endpoints.
Recommendation — Restrict query access to approved users and services with role-appropriate permissions.
NIST CSF 2.0	PR.AA-03 — Users, Services, and Hardware Are Authenticated	The proxy must validate the caller before Prometheus returns tenant-scoped results.
	PR.AA-05 — Access Permissions and Authorizations Are Managed	Tenant filtering is fundamentally an authorization problem on a shared data service.
	PR.DS-01 — Data-at-Rest Is Protected	Shared observability data can expose sensitive operational information if access is not constrained.
Recommendation — Authenticate each query caller before allowing access to metrics. Manage query permissions so each request is evaluated against the correct tenant scope. Limit exposure of stored metrics to the minimum set of authorised readers.

Practitioner Guidance

What to verify: Confirm that the proxy rejects any query that does not carry an approved tenant or permission context, and that direct Prometheus access is not reachable except through the enforcement layer. Test both normal dashboards and raw API requests, because many authorization gaps appear only outside the UI.

Decision rule: If the query can reveal data for more than one tenant or permission domain, treat the proxy as part of the security boundary and require explicit policy checks before execution. If the backend cannot enforce scope reliably, redesign the access path rather than relying on query conventions.

What good looks like: Every successful query is attributable to a caller, tied to a specific scope, and reproducibly filtered at the enforcement point. The operational goal is not to make Prometheus “aware” of every tenant model, but to ensure that shared metrics never become shared visibility by accident.

Practitioner takeaway: The safest pattern is an authorization gateway that converts caller identity and request intent into a mandatory query constraint, because tenant scoping must be enforced before results are returned, not inferred afterward.