When should organisations enrich logs in Logstash instead of waiting for Elasticsearch query time lookups?

When Ingestion-Time Enrichment Is the Better Default

Organisations should enrich logs in Logstash when the event needs a stable field that many downstream searches, alerts, and dashboards will reuse. The key advantage is that the lookup result becomes part of the stored record, so every consumer sees the same value without repeating the lookup logic at query time. That matters when the field is operationally important, such as location, ownership, or categorisation, because it reduces per-query cost and avoids inconsistent results across tools. The tradeoff is that ingestion now carries the enrichment dependency, so bad source data or stale reference data becomes harder to unwind later. For teams building repeatable detections, OWASP Non-Human Identity Top 10 is relevant when enrichment depends on machine-authenticated lookups or service access that must be governed carefully. In practice, many teams only notice the hidden cost of query-time enrichment after analysts begin running the same lookup-heavy searches at scale.

How Logstash Enrichment Changes the Search Path

Logstash enrichment shifts the lookup from the read path to the write path. Instead of asking Elasticsearch to resolve the same reference data every time a query runs, Logstash adds the field once when the event is ingested. That gives you a stored value that can be indexed, filtered, aggregated, and visualised without extra work on each search. For geolocation, asset metadata, or environment tags, this usually makes sense when the reference data changes slowly enough that the stored value remains useful for the event’s lifecycle.

The practical question is whether the enrichment result should be treated as event context or as live reference data. If it is context, ingestion-time enrichment is usually the safer and simpler choice. If it is something that must always reflect the latest source-of-truth state, query-time lookup may be more appropriate, because it avoids storing a value that can drift from current reality. That is why teams often separate “event decoration” from “authoritative lookup” rather than treating them as interchangeable.

• Use Logstash when the same field is needed across many dashboards and detections.
• Prefer query-time lookup when the reference value changes often and must remain current.
• Choose ingestion-time enrichment when search latency and analyst consistency matter more than live freshness.
• Keep the reference source small, reliable, and versioned so enrichment failures are visible quickly.

Where this guidance breaks down is when the enrichment source is volatile, unavailable, or so large that ingest-time lookups create an unreliable bottleneck.

Where the Boundary Gets Fuzzy

Tighter ingestion-time enrichment often improves consistency, but it also increases operational coupling, so organisations have to balance query speed against pipeline fragility.

One common edge case is a field that is useful for both detection and investigation but has different freshness needs in each context. In that situation, the better pattern is often to store a durable enriched field for the broad analytic use case and reserve query-time lookup for the narrower investigative workflow. Industry practice is not fully aligned on this point, because the right answer depends on how often the reference data changes and how much analysts rely on the enriched value for correlation.

Another edge case is partial enrichment. If only some events can be resolved, the resulting dataset can mislead users unless the failure mode is explicit. Teams should treat missing enrichment as a data-quality signal, not just an empty field, because silent gaps can distort dashboards and lead to false confidence in the completeness of the data. The practical judgement is not “which method is better in the abstract,” but “which method preserves trustworthy analysis for this specific field and workload.”

Practitioner Guidance

What to prioritise: Enrich at ingestion when the field is reused widely, changes slowly, and needs to behave like a durable event attribute rather than live reference data.

What to verify: Confirm how often the source data changes, how often analysts query the field, and whether stale values would create misleading detections or reporting. If the answer to staleness is “yes,” treat query-time lookup as a stronger candidate.

Decision rule: If the enrichment result is needed by many searches and the same lookup would otherwise repeat constantly, push it into Logstash; if the value must stay current above all else, keep it at query time.

Practitioner takeaway: The real decision is whether the lookup is part of the event’s meaning or part of the analyst’s question, because that determines where the operational cost and data-freshness risk should sit.

Related resources from NHI Mgmt Group

• When should organisations use time-limited access instead of standing accounts?
• When should organisations reclaim SaaS licenses instead of waiting for renewal?
• What breaks when organisations rely on audit logs instead of runtime enforcement?
• When should organisations require continuous verification instead of one-time onboarding checks?