Demand-driven kernel driver builds reduce coverage gaps and wait time

At a glance

What this is: This is an analysis of a demand-driven kernel driver build pipeline that prioritises exact-match coverage for real fleet kernels and reduces unnecessary speculative builds.

Why it matters: It matters because IAM and platform teams that manage workload identity, privileged build paths, and release pipelines need to understand how demand-driven execution changes governance, latency, and control scope.

By the numbers:

• The current batch covers roughly 540 variants across Ubuntu, Amazon Linux, Fedora, Debian, and CentOS, on both x86_64 and aarch64.
• The build service polls the GitHub Actions API every 15 seconds to find the workflow run that was dispatched after a given timestamp.
• 15 minutes before the build is marked as, e build is marked as timed out.

👉 Read Riptides' post on demand-driven kernel driver builds

Context

Kernel build coverage breaks down when the target environment is no longer a stable matrix. New kernels, live-patched variants, and distribution drift mean the right artifact may not exist when a node boots, so precompiled coverage cannot be the only control. In workload identity terms, this is a release-path governance problem as much as an engineering one: the system must decide when to reuse, when to build, and when to treat a kernel variant as new.

Riptides' approach shifts the question from "can we build everything in advance" to "can we build the exact thing the fleet needs now". That change matters for workload identity, privileged build execution, and supply-chain control because the build path itself becomes part of operational assurance. For teams managing kernel-adjacent software, the governance challenge is not simply speed. It is proving that build triggers, artifact reuse, and version matching are all bound to the actual runtime target.

The article's starting point is typical of modern fleet operations: broad prebuilds cover the common case, but the long tail still forces a just-in-time path for the outliers. That is the pattern most teams eventually hit when distribution versions, kernel sources, and deployment timing move faster than their release matrix.

Key questions

Q: How should teams handle kernel variants that are not in the prebuilt set?

A: Teams should route those misses through a demand-driven build path that first checks for an existing artifact, then dispatches a precise build only when the exact kernel, architecture, and distribution combination is absent. That keeps the fleet moving without wasting compute on speculative variants. The key control is exact-match resolution before execution, not broader precompilation.

Q: Why does distro version matter for kernel module builds?

A: Distro version matters because compiler toolchains and module metadata can differ across releases even when the kernel family is the same. A build compiled in one release may load incorrectly in another, so the artifact identity must include version, not just the distribution name. That prevents subtle runtime failures that compilation alone will not reveal.

Q: What breaks when kernel header sources age out of standard mirrors?

A: Long-tail kernels become unbuildable if the pipeline depends only on ordinary repositories. Once the headers disappear from mirrors, the build system needs fallback sources such as explicit kernel URLs or Koji and CBS. Without that chain, precompiled coverage stops at the mirror boundary rather than the fleet boundary.

Q: How should platform teams decide whether to prebuild or build on demand?

A: They should prebuild the common case and reserve on-demand execution for the long tail that the fleet actually encounters. That approach reduces wasted CI time while still preserving readiness for unusual kernels. The decision signal is real runtime demand, not how many possible variants exist in theory.

Technical breakdown

Demand-driven build dispatch

The build-service acts as a narrow control plane between the driver-loader and the GitHub Actions pipeline. It first checks whether the requested package already exists in GitHub Releases, then validates the request fields and dispatches a workflow only when needed. That pattern avoids redundant builds while preserving exact match behaviour for the kernel version, architecture, distribution, and distro version. The polling loop then reconciles the asynchronous workflow back to the caller through a build ID and status endpoint.

Practical implication: use an existence check before dispatch, otherwise every miss becomes unnecessary pipeline load.

Kernel variant resolution and fallback chains

The pipeline has to identify kernels that are no longer well represented in standard mirrors. For RPM-based distributions, that means falling back from DNF repositories to explicit kernel URLs and then to Koji or CBS when the package has rolled off the mirror. Amazon Linux adds a separate livepatch repository path, which the crawler mode must query directly because standard mirror scraping will miss those kernels. The result is a layered discovery model that maps one runtime kernel to one buildable source of truth.

Practical implication: maintain explicit fallback sources for kernel headers, or long-tail variants will become unbuildable.

Version-aligned builds and artifact keys

Driver modules are sensitive to compiler and toolchain metadata, so a build that succeeds in a different distro version can still fail at load time. The article avoids that by propagating distro_version through the request, workflow, Docker build arguments, and cache key, which keeps build environment and target environment aligned. That same field also prevents ubuntu:22.04 and ubuntu:24.04 from being deduplicated as if they were interchangeable. In practice, version is part of identity here, not just packaging metadata.

Practical implication: include distro version in cache keys and build inputs so the artifact matches the target environment exactly.

Breaches seen in the wild

• Reviewdog GitHub Action supply chain attack — reviewdog/action-setup GitHub Action supply chain attack exposed secrets.
• CI/CD pipeline exploitation case study — full server takeover via exposed .git directory and mismanaged CI/CD pipeline secrets.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Demand-driven coverage is a governance model, not just an optimisation. The article shows that speculative prebuilds reach a ceiling once kernel variance outpaces the nightly matrix. What matters is not simply building fewer artifacts, but proving that the build system can bind execution to the exact runtime target. For identity programmes, that is the same control problem seen in just-in-time privilege: scope must be resolved at the moment of need, not only at planning time. Practitioners should treat build dispatch, artifact reuse, and variant matching as part of the control surface.

Version alignment is the real control boundary. The text makes clear that "Ubuntu" is not a sufficient identity for a kernel build because toolchain differences can produce modules that load incorrectly even when compilation succeeds. That means the meaningful boundary is distro version plus kernel version plus architecture, not the distribution label alone. In governance terms, this is a reminder that coarse identity classes create hidden drift. Practitioners should model build identities at the same granularity as the runtime they serve.

Demand-driven release paths create a distinct form of identity blast radius. Once a node can trigger a build on request, the control question becomes who or what is allowed to create that demand, and how tightly the request is validated. The article's narrow API and deduplication logic limit unnecessary fan-out, but the broader lesson is that runtime requests can become privileged actions when they influence build execution. Practitioners should treat request validation and dispatch authority as part of the same governance boundary.

Just-in-time build coverage changes what "ready" means for infrastructure. The old model assumed the fleet could be made ready by precomputing every likely kernel. The new model assumes readiness emerges from exact-match artifact creation within a bounded wait window. That shift does not eliminate risk, but it makes latency and correctness explicit governance variables. Practitioners should measure build readiness against actual boot-time demand, not against the size of the prebuilt catalog.

Identity blast radius: build systems inherit the same trust problem as non-human identities. A build pipeline that decides when to create, reuse, and publish artifacts is operating as a non-human control plane. Its permissions, trigger conditions, and artifact trust chain now matter as much as its compilation logic. The implication is that workload build governance belongs in the same conversation as service account governance and privileged automation review. Practitioners should align build authority with NHI controls rather than treating it as a pure engineering concern.

From our research:

• Only 5.7% of organisations have full visibility into their service accounts, according to the Ultimate Guide to NHIs.
• 92% of organisations expose NHIs to third parties, raising concerns about supply chain security.
• Build governance and workload identity review are covered in the Ultimate Guide to NHIs, which helps teams align runtime controls with NHI lifecycle reality.

What this signals

Demand-driven build coverage points to a broader governance shift. As infrastructure becomes more variant-heavy, the control question moves from "can we precompute enough" to "can we bind execution to the exact runtime identity". That is the same logic that drives stronger workload identity and release-path controls in NIST Cybersecurity Framework 2.0 and related zero-trust designs.

Build pipelines are becoming identity-bearing systems. Once a request can trigger compilation, artifact publication, and downstream host loading, the pipeline is no longer just CI. It is a privileged non-human workflow whose request validation, dispatch logic, and cache boundaries deserve the same scrutiny as service accounts and secrets. Teams should expect build provenance and runtime readiness to be evaluated together, not separately.

Identity blast radius in this context is defined by version drift. The more closely the build environment tracks the target environment, the less room there is for hidden runtime mismatch. That is where the governance conversation should land for platform teams: measure whether build identities, artifact identities, and host identities are aligned enough to prevent avoidable failure at boot.

For practitioners

• Check for an existing artifact before dispatching a build Return the prebuilt package immediately when GitHub Releases already contains the exact kernel, architecture, distribution, and distro version combination. This reduces queue pressure and keeps the build-service from turning routine requests into unnecessary workflow runs.
• Validate the full kernel identity before triggering compilation Reject requests that do not include a well-formed kernel version, architecture, distribution, and distro version. The build input must describe the runtime target precisely enough that the artifact can be trusted at load time.
• Maintain a fallback chain for rolled-off kernel headers Use DNF first, then explicit kernel URLs, then Koji or CBS for RPM-based systems so old kernels remain buildable after standard mirrors age them out. This preserves coverage for long-tail fleet variants without maintaining a private mirror.
• Keep livepatch discovery separate from standard mirror scraping If you rely on kernel-crawler mode, query the Amazon Linux livepatch CDN directly and merge those results into the build matrix. Standard mirror scraping will miss kernels that are only published through the livepatch repository.
• Include distro version in build keys and cache logic Treat ubuntu:22.04 and ubuntu:24.04 as distinct targets even when the kernel release string looks similar. The cache key should include distro version so toolchain differences do not cause accidental deduplication.

Key takeaways

• Demand-driven kernel builds reduce waste by compiling only the exact variant a node actually needs.
• Distro version is part of artifact identity because cross-version compilation can succeed while still producing load-time failure.
• The strongest operational model combines exact-match checks, layered fallback sources, and version-aware cache keys.

Key terms

• Demand-driven build pipeline: A build pipeline that compiles only when a real runtime request proves the artifact is needed. In this pattern, the build system acts on exact kernel, distribution, and architecture identity rather than precomputing every possible variant. It improves efficiency, but it also makes request validation and dispatch authority part of the control surface.
• Kernel variant identity: The combination of kernel release, architecture, distribution, and distro version that uniquely describes the target a module must fit. Treating any one of those attributes as optional can produce artifacts that compile successfully but fail at load time. In practice, kernel identity is a release and compatibility boundary, not just a version string.
• Artifact cache key: A cache key is the identifier a build system uses to decide whether two requests are the same build or distinct work. For kernel modules, it must include every attribute that changes the resulting binary, including distro version. If the key is too coarse, the system can silently reuse the wrong artifact.
• Long-tail kernel coverage: Coverage for the uncommon kernels that appear outside the standard build matrix, such as live-patched releases, older enterprise versions, or variants that have rolled off public mirrors. The long tail is where speculative build models fail first, because the fleet eventually meets cases that were never worth prebuilding in advance.

Deepen your knowledge

Kernel identity, lifecycle, and release-path governance are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building a demand-driven platform with similar release and access constraints, it is worth exploring.

This post draws on content published by Riptides: Kernel On-Demand Driver Builds: How We Replaced Speculative Kernel Builds with a Demand-Driven Pipeline. Read the original.