Should organisations trust functional pass rates as a measure of safe AI coding?

What functional pass rates do, and what they do not prove

Functional pass rates tell you whether code behaves correctly against the tests that were written. That is useful, but it is not a security verdict. A code generator can satisfy expected outputs while still introducing injection paths, unsafe deserialization, weak authorization checks, insecure defaults, secret exposure, or brittle error handling that never appears in the functional suite.

The core limitation is coverage, not intent. Tests usually encode known requirements and expected flows, while secure implementation also depends on boundary handling, trust assumptions, data handling, dependency integrity, and failure behaviour. A high pass rate may indicate that the model followed instructions well, but it does not show that it respected the security properties the organisation actually needs.

For AI-generated code, this gap matters because many security defects are not visible in happy-path testing. The code can be functionally correct and still be unsafe under malformed inputs, privilege boundaries, concurrent access, or adversarially chosen data. That is why pass rate should be treated as one signal inside a broader acceptance decision, not as the decision itself.

Why a passing test suite can still hide security defects

Functional suites are often built to validate behaviour, not abuse cases. They tend to confirm that a function returns the right output, an API responds, or a workflow completes, but they rarely prove that the implementation resists attack paths. In practice, the missing evidence is often in negative testing, policy enforcement, and review of the generated diffs rather than in green checks alone.

This is especially important when AI coding tools can introduce subtle changes that preserve business logic but weaken security controls. For example, a model may preserve the observed output while changing input validation, broadening access, logging sensitive values, or handling credentials unsafely. Those changes can pass a task-level benchmark and still increase exploitability.

Useful supporting evidence comes from security-focused sources, not just functionality checks. NIST’s Security and Privacy Controls makes clear that access control, authentication, auditability, and configuration integrity are separate concerns from correct program output. For AI-generated code, that separation is exactly why a pass rate cannot stand in for security validation.

How to judge AI coding safely in practice

Use functional pass rate as a quality signal, but require additional evidence before promotion to production. The security question is whether the generated code still satisfies the organisation’s control expectations after compilation, integration, and deployment, not just whether it clears a unit test harness.

For AI-assisted coding workflows, the most useful companion checks are static analysis, dependency review, secret scanning, policy validation, and targeted manual review of the changed security boundary. In other words, ask whether the model changed anything that could affect trust, privilege, data exposure, or runtime assumptions. If yes, the code needs evidence beyond correctness.

External guidance on secure AI coding tools reinforces this. The OWASP Non-Human Identity Top 10 and the OWASP Agentic AI Top 10 both reflect a simple reality: automation that can write or invoke code must still be bounded by least privilege, secret hygiene, and explicit authorization. A good pass rate does not demonstrate any of those properties.

Risk and Threat Considerations

Functional pass rates can create false confidence when they are used as a proxy for safety. The risk is that teams approve code because it appears to work, then discover security regressions later through scanners, incidents, or manual review. That failure mode becomes more dangerous when AI systems generate large volumes of code faster than reviewers can inspect it.

Failure mechanism: The test suite validates intended behaviour but does not exercise malicious inputs, boundary violations, credential handling, or privilege misuse, so unsafe code can still look successful.

Impact: Organisations may ship exploitable code, miss injection or authorization flaws, and underestimate the amount of downstream remediation needed after deployment.

Practitioner Guidance

What to verify: Require at least one security-oriented control check that is independent of the functional suite, such as static analysis, secret detection, dependency review, or focused manual inspection of the changed trust boundary. If the change touches authentication, authorization, storage, or external calls, treat a passing test run as insufficient evidence on its own.

Decision rule: If the model’s output could alter privilege, data exposure, or execution path, gate acceptance on security evidence first and pass rate second. If the change is purely cosmetic or internal and cannot affect those properties, the functional signal carries more weight but still does not replace review.

Practitioner takeaway: Pass rate tells you whether the code works as written; it does not tell you whether the code is safe to trust. Use it as one input, never as the acceptance threshold.

Related resources from NHI Mgmt Group

• How should organisations measure trust across AI use cases, agents, and models?
• How can organisations tell whether an AI coding model is safe enough to use?
• Why do AI coding agents still need security gates after functional tests pass?
• What do teams get wrong about functional test pass rates in AI-assisted development?