Notifications
Clear all
Topic starter
02/08/2026 11:28 am
TL;DR: Hours-long file targeting in large repositories was traced to millions of regex calls against ignore patterns, according to Semgrep. The company cut most lookups to string comparisons and hash table checks, reducing a customer scan step from 7.5 hours to under 2 minutes and dropping p99 diff scan duration from nearly an hour to under 12 minutes; the lesson is that security tooling at scale often fails on basic path-filtering mechanics, not the detection logic itself.
NHIMG editorial — based on content published by Semgrep: file targeting performance improvements in scan optimisation
By the numbers:
Questions worth separating out
Q: How should security teams keep large repository scans fast without weakening ignore rules?
A: They should separate policy semantics from execution cost.
Q: Why do ignore patterns become a scaling problem in security scanning?
A: Because every additional file and every path prefix multiplies the number of pattern checks.
Q: What do teams get wrong about scan performance tuning?
A: They often focus on the analysis engine and overlook the pre-processing stage that decides what gets scanned.
Practitioner guidance
- Measure pre-scan filtering separately from analysis time Track file targeting, pattern matching, and scan execution as distinct stages so you can see when ignore logic rather than detection logic is consuming runtime.
- Replace generic matching with strategy-specific lookups Classify ignore patterns into literal, basename, extension, and other cheap match types before invoking regex engines.
- Index repeated pattern checks with hash tables Build dedicated indexes for each match strategy so each file only needs a small number of lookups instead of a linear walk through every pattern.
What's in the full article
Semgrep's full post covers the implementation detail this analysis intentionally leaves at the architectural level:
- The exact OCaml match-strategy logic used to replace regex-heavy path comparisons across .gitignore and .semgrepignore rules
- The before-and-after profiling data for specific repositories, including the Chromium case and the longest customer scans
- The indexing approach used to separate basename, literal, extension, and required-extension lookups at different ignore levels
- The remaining performance bottlenecks Semgrep identified, including syscall overhead in sandboxed scan environments
👉 Read Semgrep's analysis of file targeting performance and ignore-pattern optimisation →
Regex-heavy file targeting: what it means for scan performance?
Explore further
02/08/2026 12:44 pm
Scan performance is an enforcement issue, not just an engineering issue. When file targeting becomes hour-long work, teams do not experience a minor usability defect, they experience degraded security coverage. The same pattern appears in identity programmes when governance checks become too expensive to run continuously. If pre-scan controls cannot keep pace with repository scale, organisations will narrow where they enforce policy, which is how blind spots emerge in both code security and NHI governance.
A question worth separating out:
Q: How do you know when a security control is too slow to rely on?
A: When teams start reducing scope, bypassing scans, or accepting long queues as normal, the control is no longer operating as designed. A healthy control has consistent latency and a runtime profile that fits developer workflows. If enforcement is expensive, coverage will eventually erode.
👉 Read our full editorial: Semgrep file targeting shows how regex-heavy scans become slow
