How do organisations decide whether to enforce custom PromQL linting rules in CI?

Organisations should enforce custom PromQL linting when query correctness depends on local conventions that generic review will miss. Examples include required tenant labels, banned subquery forms, or approved selector patterns. If alerting mistakes would cause paging noise, missed incidents, or governance risk, CI enforcement is a sensible control.

Why This Matters for Security Teams

Custom PromQL linting is not just a style preference. It is a control decision about whether monitoring logic can be trusted before it reaches production. When teams rely on PromQL for alerting, SLOs, and operational reporting, small query mistakes can create false positives, missed detections, or noisy pages that erode confidence in the entire observability stack. That makes enforcement relevant to governance as much as engineering hygiene.

In practice, the strongest case for CI enforcement appears when an organisation has local conventions that generic linters do not understand, such as mandatory tenant labels, approved aggregation patterns, or banned use of expensive subqueries. Those conventions often reflect architecture, cost, or incident response needs. Guidance from the NIST Cybersecurity Framework 2.0 supports this kind of risk-based control selection: apply stronger checks where failure would materially affect resilience or response. In practice, many security teams encounter PromQL mistakes only after an alert storm, a silent gap in coverage, or a post-incident review rather than through intentional rule design.

How It Works in Practice

Effective enforcement usually starts with separating lint rules into two groups: universally safe syntax and organisation-specific policy. The first group covers obvious correctness issues, while the second captures rules that reflect local monitoring standards. For example, a platform team may require every selector in production alerts to include a tenant or environment label, or may prohibit queries that join across incompatible label sets. These are not universal PromQL rules, but they are often essential to accurate operation.

Teams typically implement this in CI by running a custom linter as part of pull request checks, then failing the build when a rule is violated. That approach works best when paired with clear rule ownership and documented exceptions. Common practice is to keep the linter fast, deterministic, and version-controlled alongside alerting code so reviewers can see exactly why a rule exists.

• Use hard-fail rules for issues that can break paging, routing, or compliance reporting.
• Use warning-only rules for conventions still being tested or debated.
• Scope checks to the right environment, since dev and prod often need different thresholds.
• Require explicit approval for suppressions so exceptions do not become invisible drift.

For teams already using Prometheus naming and query conventions, custom linting is the mechanism that turns those conventions into enforceable policy. Where query quality affects security operations, this also fits the broader control intent described in NIST SP 800-53, especially the parts concerned with integrity, change control, and auditability. These controls tend to break down when alert rules are generated dynamically across multiple teams because ownership, context, and exception handling become inconsistent.

Common Variations and Edge Cases

Tighter lint enforcement often increases friction for developers and platform engineers, so organisations have to balance reliability against delivery speed. That tradeoff is especially visible when custom rules encode assumptions that are only valid in one cluster, one business unit, or one observability pipeline.

Best practice is evolving here. There is no universal standard for which PromQL patterns should always fail CI, because the right answer depends on the maturity of the monitoring program and the cost of bad alerts. Some teams choose a tiered model: syntax errors and forbidden selectors fail the build, while risky but sometimes necessary patterns trigger warnings and manual review. Others permit rule exceptions only for time-limited migrations.

The hardest edge cases appear when queries are used for multiple purposes at once, such as operational alerting, capacity analytics, and compliance reporting. In those environments, one lint rule may be too blunt because a pattern that is acceptable for historical dashboards can be dangerous in paging rules. This is where governance needs to be explicit about intent, not just code quality. If the organisation also relies on shared rule libraries or generated PromQL, the same lint policy should be enforced at the source of generation, not only at the final pull request. The CISA Secure by Design guidance is relevant here because it favours preventing predictable failure modes earlier in the lifecycle. The guidance breaks down when multiple teams can bypass CI through manual hotfixes, because inconsistent enforcement quickly defeats the policy.

Related resources from NHI Mgmt Group

• How do organisations decide whether to scan Rust code in the CI pipeline or only before release?
• How should organisations decide whether to keep using traditional MFA?
• How should organisations decide whether ABAC is ready for production IAM use?
• How can organisations decide whether SPIFFE is enough for their environment?