Prometheus Security: Lock Down Endpoints, Exporters and Scrape Credentials

Prometheus, its exporters and Alertmanager ship with no authentication switched on, so a monitoring stack left reachable on a network hands out infrastructure maps, configuration and debugging data to anyone who asks. The project's own security model says untrusted users who reach the HTTP endpoint can read every time series in the database, and that /metrics, the API and the Go pprof endpoints should never be exposed to the internet. This guide covers the Prometheus server, Alertmanager, Pushgateway and the official exporters: how access works, which flags open privileged functions, where scrape credentials leak, and what a hardened baseline looks like. It is written for SRE, platform and detection engineers who run Prometheus and want a concrete checklist rather than a reminder that monitoring matters.

This is an independent guide from NHI Mgmt Group, based on the Prometheus project's public documentation as of November 2026. It is not sponsored or reviewed by the Prometheus project. Product features change, so confirm settings against the Prometheus project's current documentation.

Key takeaways

• There is no user model to fall back on. The web configuration file offers TLS, client certificates and a basic_auth_users map whose users all get full web access, so network placement and a reverse proxy carry most of the access control.
• Read access is already sensitive. /api/v1/status/config, /api/v1/status/flags and /api/v1/targets reveal the loaded configuration, start-up flags and every discovered target with its labels.
• Two flags turn reads into writes. --web.enable-admin-api exposes snapshot and series deletion, and --web.enable-lifecycle exposes /-/reload and /-/quit; both are off by default and should stay off unless a controlled caller needs them.
• Only fields marked secret are hidden. Passwords put in password, credentials or client_secret are masked, but a token pasted into a URL, label or template is served back to anyone who can read the configuration.
• Add access control to Alertmanager and Pushgateway. Without authentication in front of them, anyone with HTTP access can create silences, resolve alerts or inject series, which makes them a quiet way to blind detection.

How identity and access work in Prometheus

Prometheus is built on an explicit trust model, published on its security page. Untrusted users may reach the HTTP endpoint and logs, and can read all time series plus some operational and debugging data. Only trusted users should be able to change the command line, configuration file, rule files and other runtime settings. Scrape targets are set by configuration, but service discovery combined with relabelling hands influence to anyone who can edit data in the discovery system, such as cloud tags or Kubernetes annotations.

Built-in protection comes from a single YAML file passed with --web.config.file. Prometheus re-reads it on every HTTP request, so certificate and password changes apply without a restart. The same format, from the Exporter Toolkit, is used by Alertmanager, Node Exporter and other toolkit-based exporters.

Because the server has no per-user authorisation, teams that need read-only and admin separation, single sign-on or audit trails place an authenticating reverse proxy in front and keep the Prometheus port bound to a private interface. Default ports are 9090 for Prometheus, 9093 for Alertmanager, 9091 for Pushgateway and 9100 for Node Exporter, which is exactly what exposure scanners look for.

Admin roles and privileged access

Privilege in Prometheus is not a role, it is a flag plus network reach. Whoever can reach an enabled mutating endpoint holds that privilege, and the docs note that mutating endpoints have no built-in CSRF protection, so a browser on the same network can be tricked into calling them.

• Treat config authors as administrators: anyone who can edit prometheus.yml, rule files or the web config file can add scrape jobs, change credentials or remove alerting. Store them in version control with reviewed changes and restrict write access on the host.
• Separate the operator path: if automation needs /-/reload, allow that path only from the deployment host at the proxy, and block /api/v1/admin/, /-/quit and /debug/ for everyone else.
• Protect the host: the TSDB directory and configuration files are only as safe as the filesystem permissions and the service account running Prometheus. The docs leave on-disk protection to the operator.

For the wider pattern of controlling standing administrative paths, see the Privileged Access Management Guide.

Integrations, API tokens and service accounts

Prometheus is a credential consumer. Every authenticated scrape job, remote write target and service discovery mechanism holds a machine credential, and these are non-human identities that need the same lifecycle as any other.

• Secret-typed fields: the configuration reference marks basic_auth.password, authorization.credentials, oauth2.client_secret, oauth2.client_certificate_key, tls_config.key, proxy connect header values and http_headers secrets as <secret>. Anything else returned by /api/v1/status/config should be assumed public.
• Files over inline values: the *_file variants keep the credential out of the YAML, so the config can be reviewed and shared without carrying secrets, and the file can be mounted from a secrets manager. See the Secrets Management Guide.
• Least privilege for targets: exporters that log in to databases or brokers need a dedicated, read-only monitoring account per system, never a shared admin login.
• Probe exporters are request relays: the Blackbox and SNMP exporters take targets from URL parameters. Anyone who can reach them can point them at arbitrary hosts, and the docs warn this can leak client-side secrets such as HTTP Basic Auth passwords or SNMP community strings. Keep them reachable only from Prometheus.
• Keep TLS verification on: insecure_skip_verify exists in client tls_config; leaving it set means scrape credentials can be handed to an impersonating target.

Leaked monitoring and API tokens tend to sit unnoticed for a long time, so rotate them on a schedule (NHI Mgmt Group recommendation). For broader context on these machine credentials, see the Kubernetes NHI Security Guide.

Exporters, Pushgateway and label integrity

Exporters expand the attack surface one port at a time. Most talk to a single configured instance using preset requests, so their risk is mainly disclosure and load. The integrity risks come from components that accept data.

• Target impersonation: by default a scraped target cannot overwrite the job or instance labels Prometheus attaches; conflicting labels are renamed to exported_<label>. Setting honor_labels: true removes that protection, so restrict it to federation and Pushgateway jobs.
• Pushgateway: anyone with HTTP access can create, modify and delete metrics, and because it is usually scraped with honor_labels, they can create any series in Prometheus, including fake "healthy" signals that suppress alerts.
• Node Exporter textfile collector: it parses every *.prom file in --collector.textfile.directory. Whoever can write to that directory can publish metrics as the host, so give it tight ownership.
• Collector minimisation: use --collector.disable-defaults with only the --collector.<name> flags you need; fewer collectors means less host detail in /metrics.
• Community exporters: the project states Prometheus-Community repositories are third-party maintained and may have different security models. Review each one before deployment.

Logging and detection

Prometheus does not produce a user audit log, so detection relies on the reverse proxy, the host and Prometheus's own metrics. A useful pattern is a second, small Prometheus or the platform's log pipeline watching the first.

Map these alerts to ATT&CK so they sit in the detection backlog: external probing of exposed endpoints matches T1595 Active Scanning and T1046 Network Service Discovery, harvesting credentials from configs matches T1552.001 Credentials In Files, and silencing alerts or deleting series to hide activity fits T1685 Disable or Modify Tools. See the Grafana Security Guide for the visualisation layer that usually sits on top.

Metrics that show the controls are working:

• Number of Prometheus, Alertmanager, Pushgateway and exporter endpoints reachable from outside their allowed network, target zero.
• Percentage of monitoring endpoints served over TLS with authentication.
• Count of inline secrets found in scrape and receiver configuration, target zero.
• Silences created outside the change process per month, each reviewed.
• Days between a security release of Prometheus or an exporter and its deployment.

Hardening baseline

Roll out in an order that does not break scraping.

1. Inventory: list every Prometheus, Alertmanager, Pushgateway and exporter instance, its listen address and who can reach it. Check from outside the network for open 9090, 9093, 9091 and 9100.
2. Restrict reach first: bind to private interfaces, add firewall or network policy rules so only Prometheus reaches exporters and only the proxy reaches Prometheus and Alertmanager.
3. Add TLS in a pilot: enable tls_server_config on one exporter and switch its scrape job to scheme: https with a CA in tls_config. Confirm up stays at 1, then extend. Roll back by removing --web.config.file.
4. Authenticate: use mutual TLS with RequireAndVerifyClientCert and client_allowed_sans for machine traffic, and an SSO-backed proxy for people. Use basic_auth_users with high bcrypt cost only where neither is possible.
5. Close privileged paths: confirm --web.enable-admin-api, --web.enable-lifecycle and the receivers are off unless justified; block /debug/ at the proxy.
6. Move secrets to files: replace inline values with *_file fields, then search /api/v1/status/config output for anything that still looks like a token.
7. Secure Alertmanager clustering: set --cluster.tls-config for gossip; the docs mark this TLS support as experimental, so test upgrades.
8. Patch and verify: track releases for each component; the project targets fixes within 7 days for Prometheus, Alertmanager, Node Exporter, Blackbox Exporter and Pushgateway on a best-effort basis, and asks for govulncheck analysis rather than raw scanner output.

Common misconfigurations

• Exposed by default: a cloud instance or Kubernetes Service of type LoadBalancer publishes 9090 with no auth, giving away the configuration, target list and pprof data.
• Tokens in the wrong field: an API key placed in a scrape URL or params instead of authorization.credentials_file, so it is served by the config endpoint.
• Open Alertmanager: without authentication its /api/v2 sends Access-Control-Allow-Origin: *, so any website a user's browser visits can call the API, and anyone on the network can silence alerts.
• Blanket honor_labels: set on ordinary jobs, it lets a compromised target impersonate others and distort dashboards and alerts.
• Lifecycle enabled for convenience: --web.enable-lifecycle left on with no proxy filter, so anyone who can reach the port can stop monitoring.
• Public probe exporters: a Blackbox Exporter reachable from the internet becomes a relay that sends configured credentials to hosts of the caller's choosing.

Framework mapping

Practitioner checklist

• Confirm no Prometheus, Alertmanager, Pushgateway or exporter port is reachable from the internet.
• Serve every component over TLS 1.2 or later via --web.config.file.
• Use mutual TLS with RequireAndVerifyClientCert and client_allowed_sans between Prometheus and exporters.
• Put people behind an SSO-authenticating reverse proxy; block /api/v1/admin/, /-/quit and /debug/.
• Keep --web.enable-admin-api, --web.enable-lifecycle and the remote write and OTLP receivers off unless justified and filtered.
• Move every scrape, remote write and receiver credential to a secret-typed or *_file field and rotate any that appeared inline.
• Restrict honor_labels: true to federation and Pushgateway jobs.
• Lock down Alertmanager, review silences weekly and alert on silences created outside change windows.
• Limit Node Exporter collectors and protect the textfile collector directory.
• Track security releases for each component and patch within your vulnerability SLA.

Standards and references

• Prometheus: Security model
• Prometheus: HTTPS and authentication
• Prometheus: Configuration reference
• Prometheus: HTTP API
• Prometheus: Alertmanager HTTPS and authentication
• Prometheus: Exporter Toolkit web configuration
• Prometheus: Node Exporter
• NIST: SP 800-53 Rev. 5

Related NHI Mgmt Group resources: Grafana Security Guide · Secrets Management Guide · Kubernetes NHI Security Guide · Privileged Access Management Guide