Rust FFI sped up C# hot-path scanning by 6.5x

At a glance

What this is: Island shows that moving a sensitive-data scanning hot path from C# regex to Rust via FFI can materially reduce processing time without changing the higher-level application surface.

Why it matters: For IAM and security teams, faster content scanning affects DLP, classification, and control-plane responsiveness when sensitive-data detection sits inside user-facing or workflow-critical paths.

Context

A hot path is the part of a system that executes most often or under the tightest latency constraints, so even small inefficiencies become visible to users. In this case, the problem was not general application speed but the time it took to scan long text for sensitive data while the file was unavailable for use.

The identity and governance angle is indirect but real: sensitive-data scanning is often part of access, DLP, or workflow controls that sit alongside IAM decisions. When the scan itself becomes the bottleneck, teams end up trading security coverage against user experience unless they redesign where the work runs.

This is a performance and architecture story rather than a breach or product launch story, and the starting position is common in security-adjacent engineering: teams begin with a native library, then discover that the boundary between runtimes matters as much as the algorithm itself.

Key questions

Q: When should teams move sensitive-data scanning into a native runtime?

A: Teams should do that only when the scan path is a proven hot spot and the boundary cost is lower than the runtime gain. The decision depends on pattern complexity, file size, and how often the scanner runs in a user-facing workflow. If simple regexes already meet latency targets, the added integration complexity may not be justified.

Q: Why do scanning engines behave differently across file sizes and pattern sets?

A: Because the cost is driven by both the matching algorithm and the shape of the workload. Large pattern sets, dense match logic, and more complex expressions tend to reward engines that evaluate many patterns efficiently, while small rule sets can favour simpler in-process matching. The only reliable answer is to benchmark the exact workload.

Q: What are the main risks of crossing a managed-to-native boundary for security checks?

A: The main risks are conversion overhead, memory handling mistakes, and losing parity between the old and new enforcement logic. Teams also need to make sure the native component does not become a separate control plane with weaker testing or weaker change management. The performance gain is only useful if the scan result remains trustworthy.

Q: How should teams validate a faster scanner before replacing the original one?

A: Validate it against the same pattern corpus, the same input sizes, and the same pass or fail outcomes as the existing path. Then compare latency, stability, and edge-case behaviour for simple and complex patterns. The replacement is ready only when it preserves policy decisions and improves the cases that actually matter to the programme.

Technical breakdown

Why FFI changes regex scanning latency

Foreign Function Interface, or FFI, lets code in one runtime call functions compiled in another. Here, a thin C# wrapper loads Rust code from a DLL and passes text across the managed-native boundary for matching. The key benefit is not that Rust magically replaces all C# logic, but that the expensive part of the workload can run in a runtime with a faster regex implementation while the surrounding application stays in C#. That pattern works best when the hot path is isolated, data transfer is controlled, and the native boundary does not introduce new serialization overhead that cancels the gain.

Practical implication: move only the latency-critical scan path across the boundary, not the whole application.

Why RegexSet-style matching scales better on large pattern sets

RegexSet is a multi-pattern matching approach that checks many expressions against the same input in one pass and returns which patterns matched. That is structurally different from evaluating each regex separately, which multiplies cost as the pattern list grows. The article’s benchmark used 237 patterns, which is exactly the kind of workload where set-based matching can outperform a simpler loop over individual matches. In practice, this matters for sensitive-data scanners that need to test for many identifiers, dates, or policy keywords at once without turning every file into a long sequence of repeated evaluations.

Practical implication: use set-based matching when the detector list is broad and the same text must be evaluated many times.

Why the speedup varies with pattern complexity and file size

The benchmark results were not uniform, which is the important engineering takeaway. Smaller files and more complex patterns benefited most, while simple regexes or a small keyword set could still favour the C# implementation. That tells us the optimisation is workload-specific, not universal. Cross-runtime acceleration helps when the match logic is sufficiently heavy to outweigh the bridge cost. For security scanning, this means architecture choices should follow real pattern mix, match density, and document size rather than assuming one engine wins across every case.

Practical implication: benchmark with your own pattern mix before committing a scanning engine or language boundary.

NHI Mgmt Group analysis

Runtime boundaries are now a security-control design issue, not just an engineering detail. When sensitive-data scanning sits in the same path as user interaction or access decisions, the latency budget becomes part of the control surface. A slow detector can push teams toward narrower scanning or delayed enforcement, both of which weaken governance. The practitioner takeaway is to treat performance as a control quality property, not a separate optimisation task.

Cross-runtime acceleration is most defensible when the scanner is a bounded service, not a general-purpose refactor. The article shows a common pattern: keep orchestration in C# and push the hot path into Rust only where the gain is measurable. That is a useful model for sensitive-data detection, workload inspection, and other runtime-heavy checks that need to stay responsive. The practitioner takeaway is to isolate the hot path and measure the control boundary explicitly.

Security-adjacent scanning benefits from the same discipline as identity controls: scope, lifecycle, and test coverage. The team kept existing unit tests and used them to validate the new path, which is the right mental model for control migration. If a faster engine changes how or where policy is enforced, teams need assurance that coverage has not silently narrowed. The practitioner takeaway is to couple any performance rewrite with equivalence testing and policy regression checks.

What this signals

Scan performance has become a control-plane question for data protection programmes. When the detector sits on the path to file access or content inspection, throughput affects whether controls feel immediate enough to enforce consistently. The right design choice is not always a faster regex engine, but a narrower and better-bounded execution path for the security check.

Runtime choice now matters as much as rule quality for sensitive-data detection. Security teams should expect more mixed-language control paths where orchestration stays in the business runtime and heavy inspection moves elsewhere. The programme implication is that validation, observability, and rollback need to follow the control, not the language boundary.

Hot-path optimisation should be treated as a governance decision. If a scanning control is too slow, product teams often weaken the check or defer it, which creates silent control drift. The better approach is to test enforcement latency against user impact and keep the detection rule set aligned with the actual operating profile.

For practitioners

• Benchmark the real detector mix Measure performance using your own pattern library, file sizes, and match density before deciding whether a runtime boundary is worth adding.
• Isolate only the hot scan path Keep application orchestration in the managed runtime and move only the expensive matching logic into native code when the measured gain is material.
• Preserve policy equivalence tests Retain unit tests and add regression checks that confirm the new scanner returns the same match results as the previous implementation.
• Treat scanner choice as control design Review whether scanning speed affects enforcement timing, user blocking, or DLP responsiveness in workflows that depend on sensitive-data detection.

Key takeaways

• The article shows that sensitive-data scanning can be constrained by runtime boundaries as much as by the regex engine itself.
• The performance gains were strongest on the workloads that mattered most to the author, which makes benchmarking against real patterns essential.
• A faster scanner is only useful if it preserves policy equivalence and does not weaken the enforcement point.

Key terms

• Foreign Function Interface: A foreign function interface lets code written in one programming language call functions compiled in another. It is used to combine strengths across runtimes, but it also introduces data-marshalling, error-handling, and boundary-management concerns that must be tested like any other production dependency.
• Hot Path: A hot path is the part of a system that executes so frequently or so critically that small inefficiencies become visible to users or operational teams. In security tooling, hot paths often include scanning, matching, filtering, and policy checks that must stay fast to remain effective.
• RegexSet: RegexSet is a pattern-matching approach that evaluates many regular expressions against a single input efficiently. Instead of testing each regex separately, it groups the work so the engine can identify which patterns match in one pass, which is useful for large-scale content scanning.
• Managed-native Boundary: A managed-native boundary is the interface where code in a managed runtime, such as C#, calls into native code compiled for the operating system. It can improve performance, but it also introduces data marshaling, testing, and maintainability considerations that need explicit governance.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is suitable for teams that need to connect access governance, secrets management, and machine identity controls to operational reality.