The failure is that functional validation and security validation are not the same control. An AI agent can satisfy the requested behaviour while breaking timing safety, input sanitization, session limits, or other invariants. Teams need security tests, independent scanners, and human approval for changes that cross trust boundaries, not just green build status.
Why passing tests does not prove shipping code is safe
Tests answer a narrow question: does the code behave as expected under the cases the suite covers? Security asks a broader one: can the code be abused, bypassed, or driven into unsafe states outside the happy path? AI-generated code often passes functional checks while still introducing insecure assumptions, missing validation, or privilege mistakes that the test suite never exercises.
This is why “green build” is not a security verdict. A generated change can preserve output shape, return the right status code, or satisfy a unit test while still allowing unsafe timing, malformed input, broken session handling, or unchecked trust boundaries. The practical issue is not that tests are useless, but that they are only one control layer, and usually a weak one for adversarial behavior.
That gap is especially visible in code produced by coding agents and assistants, where the output may be syntactically correct yet semantically unsafe. The code can look finished because it compiles and passes assertions, but still fail under unexpected input, concurrent use, or attacker-controlled data. For that reason, AI-generated changes should be reviewed as security-relevant artifacts, not just productivity outputs; see NHIMG’s AI Coding Agents Security Guide for the control patterns that matter most in developer workflows.
Where the hidden failure usually sits
The most common failure is a mismatch between functional requirements and security invariants. A feature test may confirm that a request succeeds, but not that only the right user can make it succeed, that the input is safely handled, or that the operation cannot be repeated, chained, or escalated. In other words, the code can be “correct” and still be unsafe.
Another common issue is incomplete test coverage of boundary conditions. Vulnerabilities often hide in places the test author did not think to assert: parser edge cases, weak defaults, time-based behavior, trust delegation, or assumptions about upstream filtering. When AI generates code, it may reproduce those blind spots at speed, especially if the prompt optimizes for working functionality rather than defensive design.
Security validation is therefore a distinct discipline. It includes negative testing, abuse-case testing, independent scanning, and review of whether a change crosses a trust boundary. For APIs and service interfaces, broken authorization and broken authentication are classic examples of failures that normal unit tests can miss even when the endpoint “works” as designed; the OWASP API Security Top 10 is a useful reference for the kinds of failures that need explicit checks.
That distinction also applies to AI-assisted development pipelines. Anthropic’s work on secure code generation and verification, including Project Glasswing, reinforces a simple point: code quality tools can improve confidence, but they do not replace security review of the actual behavior being introduced.
How teams should validate AI-generated code before release
The right control set is layered. Start with functional tests, then add security tests that intentionally try to break assumptions, and finally require a human decision for changes that alter authentication, authorization, sessions, cryptography, secret handling, or data exposure. If a change affects who can do what, or changes how trust is established, it should not be allowed through on test pass alone.
Analysis of Claude Code Security is a useful reminder that code-generation tools can amplify both good and bad review habits. The reviewer has to verify not only whether the code meets the prompt, but whether the prompt omitted an essential constraint such as sanitization, rate limiting, or least privilege.
Practitioners should also separate scanners from approvals. A scanner can surface likely defects, but it cannot decide whether a business exception is acceptable, whether a boundary-crossing change is intended, or whether a compensation control is sufficient. That decision needs ownership, evidence, and explicit sign-off, especially for changes that touch sessions, tokens, secrets, or other identity-bearing material. NHIMG’s Sourcegraph breach 2023 is a clear example of why leaked tokens and overbroad access are not abstract risks, they are real privilege paths.
Risk and Threat Considerations
AI-generated code that passes tests but ships vulnerable logic creates a false sense of assurance. The risk is not only defect introduction, but defect invisibility, because the code can appear validated while still enabling exploit paths that the test suite never modeled.
Failure mechanism: The generated implementation satisfies the expected output under nominal conditions but fails under attacker-controlled inputs, state transitions, or trust-boundary violations, leaving authorization, validation, or session logic effectively untested.
Impact: A release can carry exploitable logic into production, expanding the blast radius from a single code defect to account takeover, data exposure, privilege abuse, or unsafe automation behavior.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Non-Human Identity Top 10 addresses the attack and risk surface, while OWASP ASVS sets the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP ASVS | V8 — Authorization | Passing tests can miss broken authorization in generated code. |
| V6 — Authentication | Unsafe generated code often breaks login, token, or session assumptions despite functional tests. | |
| V13 — Configuration | AI-generated code can ship insecure defaults or boundary settings that unit tests do not catch. | |
| Recommendation — Verify access control logic with negative tests and explicit review before release. Test authentication flows for abuse cases and rejected states, not just success paths. Check security-relevant configuration and defaults when code changes alter trust boundaries. | ||
| OWASP Non-Human Identity Top 10 | NHI-02 — Secret Leakage | Generated code can leak or mishandle secrets even when functional tests pass. |
| NHI-05 — Overprivileged NHI | AI-built code may introduce overly broad service access or privilege paths. | |
| Recommendation — Scan and review code for exposed secrets before merging any AI-generated change. Restrict service and automation privileges to the minimum required for the change. | ||
Practitioner Guidance
What to verify: Require at least one security-relevant assertion for every trust-boundary change, for example negative tests for unauthorized access, malformed input, replay behavior, and session expiry. If the change cannot be expressed as a security test, treat it as incomplete.
Decision rule: If the AI-generated change touches authentication, authorization, secrets, or input handling, route it through security review before merge, even when code coverage is high. If it only changes presentation or isolated business logic, standard review may be enough.
What good looks like: The team can explain which security invariant each test protects, which scanner findings were reviewed, and why a human approved the release. The important signal is not “tests passed,” but “the dangerous failure modes were actually exercised.”
Practitioner takeaway: Treat AI-generated code as untrusted until it has been validated against security behavior, not just functionality; the release decision should be based on whether the code is safe to abuse, not merely whether it works as requested.
Related resources from NHI Mgmt Group
- What should teams do first when AI-generated code passes tests but still looks risky?
- What breaks when AI-generated code still depends on copied AWS credentials?
- What breaks when AI-generated code reaches authentication and authorisation logic without stronger verification?
- What should teams do when AI-generated code affects authorization logic?