How should engineering teams troubleshoot intermittent CI test failures on ARM64 when the same test sometimes passes and sometimes crashes?

Start by isolating whether the failure comes from your application code or from the runtime and compiler layer beneath it. If disabling JIT for the failing function makes the issue disappear, the problem is likely in JIT code generation rather than the Lua source itself. Then capture a reproducible failing run with record and replay tooling so you can inspect the exact execution path.

Where intermittent ARM64 CI crashes usually come from

Intermittent pass-or-crash behaviour on ARM64 is rarely “just flakiness” in the abstract. It usually means the test is sensitive to a boundary condition in code generation, undefined behaviour, timing, memory ordering, or an architecture-specific runtime path. Treat the failure as a reproducibility problem first, then as a platform-specific debugging problem once you know which layer is actually unstable.

That distinction matters because a test that passes on one run and crashes on another can be masking two very different issues: a real defect in the application logic, or a compiler/runtime issue that only appears when the ARM64 execution path changes shape. The fastest way to avoid chasing the wrong layer is to compare the failing run against a controlled run with the lowest possible amount of moving parts.

On ARM64, that usually means checking whether the same binary, inputs, and environment still fail when you remove optimisation pressure or execution features that can alter generated code paths. If the crash disappears when the JIT path is disabled, or when the compiler/runtime is forced onto a simpler path, that is strong evidence that the bug is in generated code or in the runtime beneath the test rather than in the test source itself.

How to separate application defects from runtime or compiler faults

Start by narrowing the failure to one of three layers: the test logic, the runtime, or the compiler/JIT layer. If the failure reproduces only with a specific build, optimisation level, or execution mode, you are likely looking at a code-generation or runtime interaction. If it reproduces across modes but only with the same inputs, the application or test harness is the more likely source.

Record the exact environment around the failure, including compiler version, JIT or interpreter mode, CPU model, kernel, container base image, and any architecture-specific flags. On ARM64, small differences in these variables can change register allocation, instruction selection, and the timing of race conditions. That makes “works on my machine” especially unhelpful unless you can show the exact execution context that triggered the crash.

Once you have a likely failing configuration, reduce the test to a minimal case that still crashes. A good reduction keeps the same failure shape while removing unrelated setup and noise. That makes it easier to determine whether the issue belongs to the source, the generated code, or a platform dependency such as a runtime library or emulator layer.

Why record and replay is the right next move

After you have a reproducible failure path, record and replay tooling gives you a stable way to inspect the exact sequence of events that led to the crash. That is especially useful for intermittent CI failures because the bug may depend on scheduling, instruction ordering, or a transient state that disappears as soon as you rerun the job.

With replay, you can inspect the precise instruction flow, branch decisions, and data state at the moment of failure instead of relying on a fresh run that may succeed. For ARM64 issues, that can expose whether the crash is linked to a particular generated block, a bad assumption about alignment, or a runtime edge case that only appears under one execution trace.

Replay also shortens the feedback loop when you are comparing “JIT on” versus “JIT off” behaviour. If the failing trace only appears in one mode, you can focus the investigation on the code path that differs instead of broadening the search to the entire test suite. That is usually the point where intermittent crashes stop being mysterious and start becoming mechanically diagnosable.

Risk and Threat Considerations

Intermittent crashes are a reliability risk because they can hide a real defect behind non-determinism and make CI results untrustworthy. In mixed-mode runtimes, the same symptom can also indicate a memory-safety or code-generation bug that only manifests under ARM64-specific execution conditions.

Failure mechanism: Non-deterministic scheduling, JIT variability, or architecture-specific code generation changes the execution path enough to expose an otherwise latent fault, such as an invalid assumption about state, ordering, or generated instructions.

Impact: Teams can waste time debugging the wrong layer, miss a real platform defect, or ship code that only fails under the exact ARM64 conditions seen in production or CI.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

MITRE ATT&CK addresses the attack and risk surface, while NIST SP 800-53 Rev 5 and CIS Controls v8 set the governance and control requirements practitioners need to meet.

Framework	Control / Reference	Relevance
NIST SP 800-53 Rev 5	SI-10 — Information Input Validation	Intermittent crashes often stem from malformed or unexpected inputs reaching fragile code paths.
	CM-6 — Configuration Settings	Compiler flags, JIT modes, and runtime settings can change whether the ARM64 failure appears.
	AU-6 — Audit Record Review, Analysis, and Reporting	Replay traces and failing-run logs support forensic analysis of the exact failure path.
Recommendation — Validate test inputs and harness data before execution to reduce crash-triggering edge cases. Baseline and compare build and runtime settings that alter ARM64 execution behaviour. Review captured execution traces to isolate the failing instruction or state transition.
CIS Controls v8	CIS-8 — Audit Log Management	Record and replay depends on trustworthy logs and trace data for debugging intermittent failures.
Recommendation — Preserve and review trace data so intermittent crashes can be reproduced and compared.
MITRE ATT&CK	T1606 — Forge Web Credentials	No material alignment to the ARM64 CI failure subject was established, so this framework is omitted.

Practitioner Guidance

What to verify: Confirm that the crash follows the execution mode, not just the test name. If disabling JIT or simplifying the runtime path makes the issue disappear, treat that as a strong signal to investigate code generation and runtime behaviour before blaming the application test.

What to prioritise: Reproducibility beats speculation. Capture the failing binary, exact build flags, ARM64 environment, and a replayable trace before making code changes, because intermittent failures often vanish once you start probing them.

Practitioner takeaway: The most useful split is not “test versus platform” in the abstract, but “stable source defect versus execution-layer defect”, and the evidence for that split should come from a reproducible trace, not from repeated reruns.