Signals DAG architecture shows why ML pipelines need explicit dependencies

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Abnormal AI's blog post: Abnormal AI uses a Signals DAG to make ML feature extraction explicit across realtime and batch systems, with the key finding that shared dependency graphs reduce drift and improve consistency.

Why it matters: For IAM and security teams, the governance lesson is that any detection, risk, or identity pipeline that computes aggregate signals must make dependencies explicit or it will drift as systems scale and split.

Context

ML pipeline entanglement happens when changing one feature, job, or storage path unexpectedly changes downstream behaviour. In security and identity programmes, that problem shows up when realtime scoring, batch enrichment, and aggregation logic evolve separately and stop producing the same answer.

Abnormal AI frames the fix as a Signals DAG, which forces every signal extraction function to declare its inputs and outputs. That makes dependency chains visible across the pipeline, so scoring and precomputation can follow the same logic instead of diverging as volume and architecture change.

Key questions

Q: How should teams prevent drift between batch and realtime ML pipelines?

A: Teams should define a single canonical feature or signal model and reuse it across batch and realtime paths. If the same signal is implemented separately, small differences in timing, grouping, or data scope will eventually change outcomes. The governance goal is not identical infrastructure, but identical meaning for derived signals.

Q: Why do explicit dependencies matter in detection and risk scoring pipelines?

A: Explicit dependencies make it possible to see how a signal is derived and what upstream data it depends on. Without that visibility, teams cannot reliably explain why a score changed or verify that two systems are making decisions on the same basis. That becomes a governance problem as soon as the pipeline scales.

Q: What breaks when aggregate features are computed in separate systems?

A: The main failure is semantic drift. A feature can be calculated one way for offline aggregation and another way for online scoring, so the same entity receives different treatment depending on which system touched it. Once that happens, accuracy, reproducibility, and operator trust all degrade at the same time.

Q: When should organisations unify batch and streaming feature processing?

A: They should unify the logic when the same derived signal influences both realtime decisions and longer-horizon analysis. If two implementations are required for performance reasons, the dependency model still needs to stay shared. Otherwise the organisation is optimising speed while losing consistency in the underlying control plane.

Technical breakdown

Signals DAGs and explicit feature dependencies

A Signals DAG is a directed acyclic graph that maps how each signal is derived from upstream inputs. In this design, a feature is not just a calculation, it is a declared dependency with known inputs, outputs, and ordering constraints. That matters in ML pipelines because aggregate features often depend on multiple events over time rather than a single record. When those relationships are implicit, engineers end up reimplementing logic in multiple places, which increases drift and makes scoring inconsistent across services.

Practical implication: model every derived signal as a declared dependency, not a local implementation detail.

Realtime and batch feature computation are different control planes

The article shows two distinct processing modes. Realtime aggregation supports low-cardinality, time-sensitive features through streaming ingestion and rapid writes, while batch aggregation handles high-cardinality features over longer windows with more tolerance for latency. The architectural issue is not that one mode is right and the other is wrong, but that they solve different operational constraints and therefore need a common semantic model. If realtime and batch logic diverge, the same feature can mean different things depending on where it was computed.

Practical implication: keep batch and realtime feature definitions aligned, even when their execution engines differ.

Why identical DAGs reduce drift in detection pipelines

Running the same Signals DAG in scoring and aggregation systems means the feature logic used at inference time matches the logic used to precompute aggregates. That removes leaky abstractions created by parallel implementations and reduces the chance that a feature means one thing in production scoring and another in offline preparation. The article also shows that the DAG is used alongside Kafka, Spark, Redis, and Python, but the key architectural point is consistency of dependency modelling, not the choice of tools themselves.

Practical implication: reuse one dependency model across production paths so scoring and aggregation cannot silently diverge.

NHI Mgmt Group analysis

Explicit dependency modelling is becoming a governance control, not just an engineering convenience. When feature extraction logic is scattered across batch jobs, realtime services, and scoring code, teams lose the ability to reason about what a signal actually means. The Signals DAG pattern turns that hidden coupling into an auditable structure. For security and identity programmes, the lesson is that governed pipelines need declared data dependencies before they can be trusted at scale.

Separation of batch and realtime processing improves delivery speed, but it also creates semantic risk. Abnormal AI’s own description shows the tradeoff clearly: parallel systems accelerate development, yet they introduce leaky abstractions when the same signal is computed two ways. That pattern maps directly to governance problems in identity and risk systems where offline enrichment, realtime decisions, and analyst workflows drift apart. The practitioner implication is that consistency of meaning matters as much as system throughput.

Signals DAG is the right named concept for the problem this article surfaces. It is not simply a workflow graph, but a dependency contract for how signals are derived, reused, and kept consistent across environments. That concept is increasingly relevant wherever aggregate behaviour, historical context, or cross-event correlation drives security decisions. Teams should treat shared dependency definition as part of pipeline governance, not as an optional implementation detail.

Lambda-style unification is a response to leaky abstractions, not a cosmetic architecture choice. The article makes clear that separate systems create maintenance overhead and semantic mismatch even when they scale independently. Unification only helps if the underlying dependency model remains stable across execution modes. For practitioners, the real decision is whether their pipeline architecture preserves one source of truth for derived signals across operational contexts.

Tool choice is secondary to semantic control in high-scale detection systems. Kafka, Spark, Redis, and Python matter here because they support scale, but they do not solve the core governance issue on their own. The central risk is uncontrolled divergence in how signals are computed and consumed. Organisations building detection or identity analytics pipelines should focus first on dependency fidelity, then on platform selection.

What this signals

Dependency visibility is now part of pipeline trustworthiness. As ML and detection systems spread across realtime and offline execution paths, governance teams need one canonical model for how signals are derived and consumed. Without that, operating teams end up reconciling behaviour instead of governing it, which is a structural weakness in any security analytics programme.

Leaky abstractions are the real scaling tax. The more a pipeline splits into specialised services, the more likely identical business logic will diverge across implementations. For identity, risk, and detection programmes, that means consistency checks belong alongside performance monitoring, because the failure mode is semantic drift rather than simple outage.

For practitioners

• Define signal contracts before scaling pipelines Document each derived signal’s inputs, outputs, and dependency order so scoring and aggregation use the same semantics across environments.
• Unify realtime and batch feature logic Review where batch jobs and streaming services compute the same feature differently, then standardise the derivation logic across both paths.
• Treat drift as a production defect Add checks that compare realtime and offline feature outputs for the same entity, then investigate any mismatch as a governance failure.
• Separate execution engines from dependency meaning Allow Kafka, Spark, Redis, or other engines to differ operationally, but keep one canonical definition for each signal and aggregate.

Key takeaways

• The article shows that explicit dependency modelling is the mechanism that keeps complex ML pipelines from drifting as they split across execution environments.
• It also shows that batch and realtime systems can both be useful, but only if they preserve one definition for each derived signal.
• For security and identity practitioners, the practical lesson is to govern signal meaning centrally before optimising for scale or speed.

Key terms

• Signals DAG: A Signals DAG is a directed acyclic graph that maps which data sources, enrichments, and downstream decisions depend on one another. In security scoring, it makes hidden dependencies visible so teams can reason about what fails, what is quarantined, and what still deserves trust when upstream services are unavailable.
• Aggregate Feature: An Aggregate Feature is a derived signal built from patterns across multiple events rather than from a single observation. In security and identity analytics, these features often reveal behaviour that is invisible at message or request level, but they also require careful precomputation and governance.
• Leaky Abstraction: A leaky abstraction occurs when implementation details from one layer of a system become visible to users of another layer, forcing them to manage hidden complexity manually. In detection pipelines, that usually means separate systems behave differently enough that engineers must keep reconciling their outputs.
• Lambda Architecture: Lambda architecture combines batch and realtime processing paths so systems can support both speed and completeness. The model is useful when teams need low-latency decisions and longer-horizon recomputation, but it only works well if both paths preserve the same signal definitions.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.