Scrape Discovery

The mechanism Prometheus uses to find targets automatically, often by reading Kubernetes annotations and labels. It simplifies operations, but it also means collection boundaries depend on cluster metadata and policy discipline, not only on Prometheus configuration.

Expanded Definition

Scrape discovery is Prometheus target discovery that automatically finds scrape endpoints from cluster metadata, commonly Kubernetes labels and annotations. It reduces manual upkeep, but it also makes monitoring scope dependent on how consistently teams apply metadata and policy.

In NHI and agentic environments, scrape discovery matters because it can reveal more than metrics endpoints. It can surface service names, namespaces, ports, and sometimes labels that hint at workload purpose or trust boundaries. That means the mechanism is operationally convenient, yet also sensitive to metadata quality and exposure control. The security model is not defined by Prometheus alone; it is shaped by cluster governance, identity boundaries, and how workload metadata is managed. Guidance varies across vendors, but the practical principle is consistent: discovery should be treated as a controlled intake path, not an open invitation to monitor everything that appears in the cluster. The Prometheus scrape_config documentation describes the discovery model in operational terms, while NHI governance sources such as the NHI Lifecycle Management Guide frame the broader lifecycle and control expectations.

The most common misapplication is treating annotation-driven discovery as equivalent to authorised observability, which occurs when metadata is allowed to define monitoring scope without review.

Examples and Use Cases

Implementing scrape discovery rigorously often introduces tighter metadata discipline, requiring organisations to weigh monitoring agility against the risk of accidental exposure.

• A Kubernetes cluster uses labels to auto-register application pods, so new replicas become observable without editing Prometheus configuration.
• A platform team restricts which namespaces can be discovered, using annotations only for approved workloads and pairing the rule with Kubernetes RBAC.
• A security team reviews discovered targets against OWASP NHI guidance to ensure service identities are not being surfaced outside intended boundaries.
• An agentic AI deployment exposes internal metrics endpoints through labels, then uses Top 10 NHI Issues to evaluate whether the same metadata could help attackers map privileged services.
• An observability engineer adds admission controls so only workloads with approved annotations are eligible for discovery, aligning with the cluster policy approach described in the Ultimate Guide to NHIs.

In practice, scrape discovery is also used to avoid brittle static target lists when workloads scale rapidly or are recreated frequently. That convenience is valuable in environments where service accounts, API keys, and short-lived pods change often, but the discovery rules must still be auditable and limited to approved metadata sources. The Kubernetes service discovery documentation shows how metadata becomes the discovery substrate.

Why It Matters in NHI Security

Scrape discovery matters because observability metadata can become an attack surface. If discovery is too broad, Prometheus may learn about workloads that were never intended for routine collection, and that visibility can help an attacker map service relationships, target sensitive namespaces, or infer which non-human identities are active. The risk is not just disclosure. It can also create false trust in what has been “covered” by monitoring, when in fact coverage was granted by labels that were never reviewed for security impact.

This is especially relevant in NHI governance because identity sprawl and visibility gaps frequently travel together. NHI Mgmt Group reports that only 5.7% of organisations have full visibility into their service accounts, which means discovery mechanisms often become a de facto inventory source even when they were never designed as one. That makes metadata governance, least privilege, and review of discovered targets critical to safe operation. The same logic aligns with the EU Cyber Resilience Act perspective on secure-by-design lifecycle controls, and with the Ultimate Guide to NHIs emphasis on lifecycle visibility and governance.

Organisations typically encounter the consequences only after an unintended target is discovered, at which point scrape discovery becomes operationally unavoidable to address.

Related resources from NHI Mgmt Group

• Why is NHI discovery and inventory the primary goal of NHI security?
• Why is continuous discovery of AI agents important?
• When does secrets discovery become insufficient on its own?
• When should organizations consider adopting advanced tool discovery for AI agents?