Semgrep file targeting shows how regex-heavy scans become slow

At a glance

What this is: Semgrep says its file targeting step could spend hours matching ignore patterns in large repositories before the team replaced most regex lookups with string comparisons and hash indexing.

Why it matters: For practitioners, this shows how scan performance, control coverage, and developer trust all depend on the efficiency of pre-scan filtering, not just the quality of findings.

By the numbers:

• the 99th percentile diff scan duration dropped from nearly an hour to under 12 minutes

👉 Read Semgrep's analysis of file targeting performance and ignore-pattern optimisation

Context

File targeting is the part of a scan that decides which files should be included before analysis begins. In Semgrep’s case, the issue was not the scanning engine itself but the cost of repeatedly evaluating ignore patterns across large codebases, which turned a simple filtering problem into a performance bottleneck.

This matters to application security and identity governance teams because slow, inconsistent pre-processing weakens developer adoption and can delay enforcement in the same way that poor lifecycle controls weaken NHI governance. When security tooling cannot process large repositories quickly, teams often narrow scope, skip scans, or tolerate stale configuration, which creates blind spots across code, secrets, and the systems that manage access.

Semgrep’s starting position is not unusual for large-scale security tooling: once repositories get big enough, path filtering and pattern matching can dominate runtime even when the underlying security checks are efficient.

Key questions

Q: How should security teams keep large repository scans fast without weakening ignore rules?

A: They should separate policy semantics from execution cost. Use cheap, strategy-specific matching for common ignore patterns, then reserve regex for the smaller set of cases that truly need it. The goal is to preserve scan coverage while eliminating unnecessary repeated work that turns path filtering into the performance bottleneck.

Q: Why do ignore patterns become a scaling problem in security scanning?

A: Because every additional file and every path prefix multiplies the number of pattern checks. When those checks rely on regex by default, even simple exclusions can become millions of operations in large repositories. At scale, that overhead delays enforcement, increases compute cost, and makes security controls harder to trust.

Q: What do teams get wrong about scan performance tuning?

A: They often focus on the analysis engine and overlook the pre-processing stage that decides what gets scanned. In many large environments, file selection, path filtering, and ignore evaluation consume more time than the security checks themselves. Fixing the hot path before analysis often produces the biggest gains.

Q: How do you know when a security control is too slow to rely on?

A: When teams start reducing scope, bypassing scans, or accepting long queues as normal, the control is no longer operating as designed. A healthy control has consistent latency and a runtime profile that fits developer workflows. If enforcement is expensive, coverage will eventually erode.

Technical breakdown

Why ignore-pattern filtering becomes a scan bottleneck

Semgrep’s file targeting step has to evaluate candidate paths against ignore rules from .gitignore, .semgrepignore, backend configuration, and defaults. When that is implemented as repeated regex matching across every file path and every relevant prefix, the number of comparisons grows very quickly. The cost is not just algorithmic complexity in theory. In practice, large repositories create millions of pattern checks, and each check adds CPU time before any security analysis begins. That means the scan is delayed by bookkeeping rather than by detection logic. For teams running security checks in CI, this is a classic control-plane failure: the enforcement stage is too expensive to execute consistently.

Practical implication: reduce repeated pattern evaluation before scan execution, or the control will become too slow to trust.

How string comparisons changed the matching model

Semgrep improved the hot path by recognising that many ignore patterns are not really regex problems at all. Literal, basename, extension, and required-extension cases can be evaluated with direct string equality or a small amount of pre-check logic. That shifts the work from a general-purpose regex engine to simple path classification. The important technical point is that the optimisation preserves semantics while removing unnecessary computation. This is the same kind of design shift that often matters in identity systems too: if a decision can be resolved by a deterministic lookup instead of a heavyweight evaluation path, runtime and failure surface both improve.

Practical implication: classify the common case into cheap match types before invoking expensive pattern engines.

Why hash tables matter for large-scale path matching

Once ignore rules are reduced to a small set of string-shaped match strategies, the next problem is searching those patterns efficiently. Semgrep borrowed the same core idea used by fast matching systems: build indexed lookups so the scanner can check many candidate patterns in constant time rather than walking a flat list for every file. Separate tables are needed for different match strategies because correctness depends on comparing like with like. That is why the improvement scales: the scanner still respects ignore semantics, but it no longer pays linear cost for every path-pattern pair. For high-volume security tooling, this is the difference between acceptable overhead and operational drag.

Practical implication: use strategy-specific indexes for ignore evaluation so scaling file counts do not translate into linear runtime growth.

NHI Mgmt Group analysis

Scan performance is an enforcement issue, not just an engineering issue. When file targeting becomes hour-long work, teams do not experience a minor usability defect, they experience degraded security coverage. The same pattern appears in identity programmes when governance checks become too expensive to run continuously. If pre-scan controls cannot keep pace with repository scale, organisations will narrow where they enforce policy, which is how blind spots emerge in both code security and NHI governance.

Path filtering is a hidden source of security debt. Most practitioners focus on the detection engine, but this case shows that the gating logic before analysis can be the real bottleneck. The named concept here is filtering overhead debt: the accumulated runtime and complexity cost of enforcing simple policy decisions through overly general mechanisms. Once that debt grows, teams start treating enforcement as optional because it is operationally painful. The practical conclusion is that governance should be designed for cheap repetition, not one-time correctness alone.

Semantics-preserving optimisation is the right model for scale. The useful design move here was not to weaken ignore behaviour, but to recognise which cases could be resolved through direct comparison and which still required regex. That is a broader lesson for security architecture: preserve policy intent while changing the execution path. In identity and NHI controls, the same principle applies when routine decisions can be moved from costly review loops to indexed, deterministic checks. The practical conclusion is to redesign enforcement paths around scale, not around the assumption of small repositories.

Developer-facing security tools have to be fast enough to disappear into the workflow. If scanning adds hours, the organisation eventually feels pressure to reduce coverage, defer scans, or route around the control. That is dangerous because speed is part of control efficacy, not an extra. For teams governing code security, secrets exposure, or identity-related automation, runtime variability is a sign that the control plane is becoming unpredictable. The practical conclusion is to treat latency thresholds as governance requirements, not just operational metrics.

What this signals

Filtering overhead debt: when enforcement logic becomes expensive to run, security teams quietly reduce coverage and normalise exceptions. That pattern matters beyond code scanning because any control that cannot execute cheaply enough will eventually lose policy authority. For identity programmes, the lesson is the same: lifecycle checks, access reviews, and secrets governance must be designed for constant repetition, not occasional success.

Teams should watch for controls whose runtime grows faster than repository size, access volume, or policy complexity. That is usually the point where governance starts drifting into advisory mode. In identity and NHI programmes, the practical response is to move from repeated heavy evaluation to indexed state, narrower match sets, and clearer enforcement boundaries.

Where identity-relevant automation is involved, performance is not just an engineering metric. It shapes whether the control is actually used, whether exceptions proliferate, and whether developers trust the security gate. The next step is to treat latency as a governance indicator alongside coverage, false positives, and remediation time.

For practitioners

• Measure pre-scan filtering separately from analysis time Track file targeting, pattern matching, and scan execution as distinct stages so you can see when ignore logic rather than detection logic is consuming runtime. Use the slowest repositories as the primary signal, not fleet averages.
• Replace generic matching with strategy-specific lookups Classify ignore patterns into literal, basename, extension, and other cheap match types before invoking regex engines. That reduces repeated work and makes the cost of path filtering predictable at scale.
• Index repeated pattern checks with hash tables Build dedicated indexes for each match strategy so each file only needs a small number of lookups instead of a linear walk through every pattern. This is especially important when repositories contain deep directory trees or many default ignores.
• Set latency thresholds for security controls Treat scan duration SLOs as part of control design and flag any step that regularly exceeds the threshold as a governance issue. If a control is too slow, teams will eventually work around it.

Key takeaways

• Semgrep’s file targeting issue shows that security controls can fail operationally long before they fail logically.
• Millions of repeated pattern checks turned a simple ignore step into hour-long scan latency for large repositories.
• The practical fix is to preserve policy semantics while replacing linear matching with cheaper, indexed lookups.

Key terms

• File Targeting: The pre-scan step that decides which files should be included in analysis. It applies ignore rules and path filters before any security checks run, so its performance directly affects whether the overall control feels usable at scale.
• Ignore Pattern: A rule that excludes files or paths from scanning based on names, prefixes, suffixes, or other path characteristics. In code security tools, ignore patterns protect signal quality but can become expensive if evaluated inefficiently across large repositories.
• Match Strategy: The specific way a pattern is evaluated against a file path, such as direct string equality, basename matching, or extension matching. Strategy-aware matching reduces unnecessary regex calls and is a common technique for keeping large-scale filtering fast.

What's in the full article

Semgrep's full post covers the implementation detail this analysis intentionally leaves at the architectural level:

• The exact OCaml match-strategy logic used to replace regex-heavy path comparisons across .gitignore and .semgrepignore rules
• The before-and-after profiling data for specific repositories, including the Chromium case and the longest customer scans
• The indexing approach used to separate basename, literal, extension, and required-extension lookups at different ignore levels
• The remaining performance bottlenecks Semgrep identified, including syscall overhead in sandboxed scan environments

👉 Semgrep's full post covers the trace data, match-strategy changes, and repository-level timing results

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and identity lifecycle controls. It helps practitioners connect access governance to the operational realities that shape security outcomes.