Why do functionality tests matter before security tests in code repair benchmarks?

Why functionality has to pass before security hardening

In a code repair benchmark, functionality is the floor, not the finish line. If a patch breaks compilation, alters expected behavior, or fails the original use case, you no longer know whether you repaired the bug, introduced a new defect, or simply changed the program enough to make later security checks unreliable.

That ordering matters because security testing only has meaning against working software. A benchmark that checks behavior first can separate a real fix from a patch that merely removes the visible symptom while damaging correctness, which is especially important when repair quality is being compared across submissions.

A strong benchmark therefore treats syntax, execution, and expected outputs as the first gate. Only after the candidate repair preserves the intended behavior does it make sense to ask whether the repair also closes the security weakness or leaves a new attack path behind.

What security testing can and cannot tell you after a broken fix

Security tests are evidence about exposure, not a substitute for correctness. If the repaired code no longer runs or produces the wrong result, a passing security check does not prove much, because the system may have escaped the vulnerable path by accident rather than by design.

This is why repair evaluation often needs two distinct questions: “Does the code still work?” and “Is it still secure?” The first question protects against false positives in the second. Without that separation, a benchmark can reward changes that look safer simply because they disabled functionality, narrowed execution, or short-circuited the code path under test.

Functionality-first evaluation also improves comparability. When every candidate fix is required to preserve the same baseline behavior, security scores become easier to interpret because they reflect the effect of the repair itself, not the side effects of a broken implementation.

What benchmark designers should optimize for in repair evaluation

Benchmark design should make regressions obvious and cheap to detect. That usually means running compile, unit, or expected-behavior checks before any deeper security analysis, then using security tests to distinguish a correct repair from a merely functional one. In practice, this avoids rewarding patches that satisfy the security oracle while silently degrading the original program contract.

The same principle applies to scoring. A repair that fixes the vulnerability but fails baseline tests should not outrank one that preserves behavior and reduces risk. For practitioners, the useful benchmark is the one that captures both dimensions in order, so correctness remains a prerequisite for security rather than a competitor to it.

Risk and Threat Considerations

When security checks run before correctness checks, teams can misclassify a broken patch as a secure repair. That creates a control blind spot: the benchmark may approve changes that are safer only because they no longer execute the intended logic, which can hide both functional regressions and residual exposure.

Failure mechanism: The repair changes control flow, validation logic, or error handling enough to defeat the vulnerable path while also breaking the original behavior, so the security signal is no longer anchored to a valid application state.

Impact: Evaluators may ship or reward fixes that are not actually usable, while attackers or test bypasses can exploit the gap between “looks secure” and “still behaves correctly.”

Practitioner Guidance

What to verify: Require a clean functional baseline before interpreting any security result. If the repaired code does not compile, fails core tests, or changes expected output, treat the security score as provisional rather than meaningful.

Decision rule: If a patch improves security but degrades required behavior, classify it as an incomplete repair, not a successful benchmark outcome. The acceptable fix is the one that preserves intended functionality and then reduces the vulnerability.

Practitioner takeaway: In repair benchmarks, correctness is the trust anchor, and security testing only becomes useful once that anchor is intact.

Related resources from NHI Mgmt Group

• How should security teams discover AI usage in source code before deployment?
• What should security teams do before connecting AppSec output to code?
• How should security teams validate appsec findings before fixing code?
• How should security teams validate AI-generated code fixes before they are merged?