Dynamic garbage collection tuning for memory-heavy OCaml workloads

At a glance

What this is: This is a performance engineering post showing that OCaml 5 memory regressions in a memory-heavy Semgrep workload were caused by major GC behaviour, and that dynamic tuning of space_overhead restored practical memory use.

Why it matters: It matters because teams upgrading runtimes or scaling analysis workloads need to govern memory behaviour as an operational control, not assume defaults will hold across repository sizes or execution profiles.

Context

OCaml 5 changed the runtime model enough that a workload with heavy analysis and large allocation footprints no longer behaved like the same code on OCaml 4. In practice, that means the memory profile of an application can shift even when the application logic stays stable. For identity and security teams running analysis platforms, the lesson is familiar: control assumptions based on one workload shape often fail when the environment or scale changes.

Semgrep's experience is a runtime governance problem rather than an application bug story. The article shows that static garbage collector settings were adequate for some repositories and too conservative or too aggressive for others, which is exactly the kind of mismatch that creates operational instability in memory-sensitive services. The broader implication is that platform teams need tuning strategies that adapt to actual workload behaviour instead of freezing one parameter for every scan.

The article is also a reminder that memory pressure has security-adjacent effects. Resource exhaustion, crashes, and unstable throughput are availability failures, but they also complicate scanning, analysis, and other control-plane tasks that security teams depend on. Where tools sit inside a wider identity or security programme, runtime efficiency becomes part of control reliability.

Key questions

Q: Why do runtime upgrades sometimes increase memory use on the same workload?

A: Runtime upgrades can change garbage collection and allocation behaviour even when the application code does not change. If the new runtime reclaims memory less aggressively, or if compaction and allocator behaviour differ, the same workload can hold more live data longer and push the process footprint higher. Teams need to validate memory behaviour after upgrade, not assume feature compatibility means operational parity.

Q: How should teams choose GC settings when workload sizes vary widely?

A: Teams should tune garbage collection against representative workload classes instead of one median case. Small jobs may benefit from less aggressive collection for speed, while large jobs often need tighter memory control to avoid exhaustion. The goal is to set policies that adapt to heap growth or workload stage, rather than locking in one static threshold for every execution.

Q: What are the signs that a memory regression is caused by GC behaviour?

A: A GC-driven regression usually shows up as higher RSS, slower reclamation of dead values, and a gap between live allocation traces and the process footprint the operating system sees. If compaction, allocator changes, or code changes do not explain the increase, GC aggressiveness is a likely control to investigate. Measure both runtime traces and system memory together.

Q: When should organisations use adaptive runtime controls instead of static defaults?

A: Use adaptive controls when workload shape, repository size, or traffic patterns vary enough that one default produces contradictory outcomes. In that situation, a single value forces a tradeoff between speed and memory that may be acceptable for one job and harmful for another. Adaptive tuning keeps the control aligned to current runtime conditions instead of stale assumptions.

Technical breakdown

Why OCaml 5 changed memory behaviour

OCaml 5 introduced multicore support, which required a substantial runtime redesign, especially in the garbage collector and allocator. The key issue is that memory behaviour is not just about how much the application allocates. It also depends on when the runtime reclaims dead values, how much heap it keeps around, and whether unused memory returns to the operating system. In this case, the same workload that behaved acceptably on OCaml 4 consumed far more memory on OCaml 5 because the major GC was less aggressive under the default settings for that workload shape.

Practical implication: treat runtime upgrades as memory-behaviour changes, not just feature changes.

What Memtrace and RSS reveal about GC tuning

Memtrace shows live and collected allocations over time, which helps distinguish between actual allocation growth and delayed reclamation. RSS shows the process footprint the operating system sees, including heap waste and fragmentation. Using both matters because a runtime can look acceptable in one view and still exhaust machine memory in the other. In this article, the combination showed that major GC behaviour, not just allocator differences or compaction gaps, explained the regression. That is a useful pattern for teams diagnosing performance drift after runtime changes.

Practical implication: measure both live memory and RSS before deciding which runtime control failed.

How dynamic garbage collector tuning works

Dynamic tuning in this case uses GC alarms to run logic at the end of each major collection and GC stats to inspect the major heap size. The utility then adjusts space_overhead by interpolation, making the collector less aggressive for smaller heaps and more aggressive as the heap grows. That matters because static settings were a poor fit across repository sizes: values that protected memory on large scans could slow smaller scans, while faster settings could blow out memory on larger ones. The control point becomes adaptive rather than fixed.

Practical implication: use workload-aware GC policies when one static threshold cannot satisfy all execution profiles.

Threat narrative

Attacker objective: The operational objective was not adversarial compromise but restoring stable execution so the analysis workload could run without exhausting memory or crashing.

1. Entry occurred when Semgrep upgraded a memory-heavy analysis workload from OCaml 4 to OCaml 5 and inherited new runtime behaviour that the old benchmarks did not capture.
2. Credential or privilege abuse is not the issue here; the failure mode was delayed major garbage collection and insufficient reclamation under default settings.
3. Escalation happened as heap growth and process RSS climbed until benchmarks hit resource exhaustion and crashes on larger repositories.
4. Impact was degraded reliability and failed scans, which blocked a safe production rollout until GC tuning reduced memory pressure.

NHI Mgmt Group analysis

Runtime memory governance is now part of security tooling reliability. When analysis engines, scanners, or policy services move to newer runtimes, the main failure mode is often not functional breakage but resource instability. That instability can degrade the controls downstream teams rely on, especially when a tool must finish work under bounded memory. The practitioner takeaway is to treat runtime tuning as a control-plane decision, not a purely engineering detail.

Dynamic memory control is a better pattern than static tuning when workload size varies. The article shows that one GC setting could not serve small and large repositories equally well, which is a common shape in real security programmes. Static thresholds create false confidence because they assume a stable workload distribution. Practitioners should expect runtime parameters to be environment-sensitive and govern them accordingly.

Performance regressions after platform upgrades often reveal hidden control assumptions. The interesting point here is not that OCaml 5 changed memory use, but that the default runtime assumptions were not aligned to one production workload. That is the same class of problem identity and security teams see when defaults, policy engines, or scanners meet large, uneven estates. The practitioner conclusion is to validate runtime behaviour against real workloads before rollout.

Measure the control that actually affects availability, not the control that is easiest to inspect. Memtrace, RSS polling, and GC stats each exposed a different part of the problem, and only together did they justify the final tuning approach. This is the right mindset for control validation in any operational security platform: observe the resource that fails first, not the one that is most convenient to chart.

One-size-fits-all tuning is a governance anti-pattern. The article's final position is that user-tunable or workload-aware controls are more realistic than a universal default when consumption patterns vary materially. That does not remove the need for safe defaults, but it does mean the default should not be treated as a policy boundary. Practitioners should plan for bounded flexibility rather than rigid uniformity.

What this signals

Runtime tuning becomes a governance issue once a platform's behaviour changes with workload shape. Security and analysis teams often inherit runtime defaults as if they were neutral, but the article shows that the default can become the limiting control when scale changes. For practitioners, that means rollout plans should include memory validation and not just functional tests.

Adaptive control is the right mental model for memory-heavy analysis pipelines. The underlying lesson is broader than OCaml: if a workload varies enough to cross performance and memory thresholds, static configuration stops being reliable. Teams should design for monitored adjustment rather than expecting a single universal setting to hold across all environments.

For practitioners

• Profile live memory and process RSS together Use both allocation tracing and operating-system memory polling when validating runtime changes so you can separate delayed collection from true footprint growth.
• Test runtime upgrades against representative workload sizes Benchmark small, medium, and large repositories separately, because the GC setting that works for one scan shape may degrade either latency or memory at another scale.
• Treat GC parameters as production controls Document which runtime parameters affect availability, assign ownership for them, and review them after language or runtime upgrades just as you would other capacity-sensitive settings.
• Prefer adaptive tuning over a single static threshold If workload size varies materially, use a tuning policy that changes with heap or memory behaviour instead of freezing one space_overhead value for all runs.

Key takeaways

• OCaml 5 introduced a memory-behaviour change that exposed the limits of static GC assumptions in a real analysis workload.
• The practical evidence came from both trace data and RSS, which showed that memory pressure was not just a compaction or allocator artefact.
• Adaptive garbage collector tuning restored acceptable behaviour, which makes runtime governance a rollout requirement for memory-heavy platforms.

Key terms

• Garbage Collector Tuning: Garbage collector tuning is the practice of changing runtime parameters so the collector reclaims memory more or less aggressively. In long-running systems, the goal is to balance throughput, latency, and memory footprint without changing application logic.
• Resident Set Size: Resident set size is the amount of physical memory a process is currently using on the host. It includes live data and overhead from the runtime, allocator, and heap behaviour, so it is often the best measure of real operational cost.
• Disk Fragmentation: Disk fragmentation occurs when files grow into non-contiguous blocks across storage, forcing extra read and write work. In database and identity workloads, fragmentation can slow processing, increase I/O overhead, and make growth behavior less predictable as files expand over time.
• Dynamic Runtime Control: Dynamic runtime control is an approach where a system changes internal parameters while it runs, based on measured conditions rather than fixed defaults. For memory-heavy workloads, this can balance speed and footprint more effectively than a single static setting across all job sizes.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. For practitioners who own identity, access, or machine-credential programmes, the course helps connect governance to operational control.