AI-Generated Code Review Guide: Catching Flaws Before They Ship

AI coding assistants now write a large share of the code that reaches pull requests, and review processes built for human authors miss the specific ways that code goes wrong. Generated code compiles, passes the happy-path test and reads fluently, which is exactly why reviewers wave it through. Research since 2021 has shown repeatable weaknesses: missing input handling, unsafe defaults, invented dependencies and authorisation checks that are simply absent. This guide sets out how to review AI-generated code: what to look for, which automated gates to put in front of the human reviewer, how to track provenance, and how to measure whether the process is working. It is written for application security engineers, engineering leads and platform teams who own the pull request workflow.

Key takeaways

• Fluent is not the same as safe. Independent studies found that around 40% of programs generated in security-relevant scenarios were vulnerable, and that developers using an assistant were more confident in less secure code.
• Review the change, not the author. Apply the same or stricter bar to AI-written code, with a checklist tuned to its typical failures: absent authorisation, unvalidated input, invented packages, weak cryptography and leaked secrets.
• Put machines in front of humans. Secret scanning, dependency existence and reputation checks, SAST and tests should run before a reviewer sees the diff, enforced as required checks rather than advice.
• Record provenance. Mark AI-assisted commits and pull requests so you can route them, sample them and investigate them later, and so an unattributed agent pull request cannot merge on a single approval.
• Instruction files are code too. Rules files such as CLAUDE.md, .cursorrules and Copilot instruction files steer what the assistant writes, so changes to them need code owner review.

Why AI-generated code needs a different review lens

Human reviewers have learned to spot the mistakes humans make: typos, copy-paste errors, half-finished refactors. AI output fails differently. It is syntactically clean, idiomatic and confident, and its flaws are usually omissions rather than visible errors. A reviewer scanning for something that looks wrong will not see a missing ownership check on an API handler.

• Measured vulnerability rates: Pearce and colleagues generated 1,689 programs across 89 scenarios based on high-risk CWEs, including the CWE Top 25, and found about 40% vulnerable (IEEE Symposium on Security and Privacy 2022). Veracode reported in July 2025 that across more than 100 LLMs and 80 coding tasks, generated code introduced a vulnerability in 45% of cases, with Java above 70%; these are the vendor's own figures.
• Specific weak classes: the same Veracode work reported that models failed to defend against cross-site scripting (CWE-80) in 86% of relevant cases and log injection (CWE-117) in 88%. Output encoding and log neutralisation therefore deserve explicit review attention.
• Overconfidence: Perry, Srivastava, Kumar and Boneh found that study participants with an AI assistant wrote significantly less secure code and were more likely to believe it was secure. Participants who trusted the assistant less and reworked their prompts did better. The reviewer is the control that compensates for that overconfidence.
• Invented dependencies: Spracklen and colleagues analysed 576,000 code samples from 16 models and found hallucinated package rates of at least 5.2% for commercial models and 21.7% for open-source models, with 205,474 unique invented names (USENIX Security 2025). Attackers can register those names. See the Slopsquatting and Package Hallucination Guide.
• Volume: assistants and agents produce larger diffs faster than people do. Review capacity, not code generation, becomes the constraint, and tired reviewers approve more.

OWASP's Top 10 for LLM Applications 2025 lists unsafe code generation, including suggestions of insecure or non-existent libraries, under LLM09 Misinformation, and treats overreliance on unverified output as the core risk. The answer is not to ban the tools but to make verification systematic.

What to look for: a review checklist by flaw class

Use this table as the reviewer's prompt sheet for any pull request marked as AI-assisted. The CWE column helps map findings to SAST rule packs and the CWE Top 25 Guide.

Two further checks are specific to generated code. First, test integrity: assistants asked to make tests pass sometimes weaken or delete assertions, mock the thing under test, or add skip markers. Review test diffs as carefully as production diffs. Second, scope creep: agents often touch files outside the stated task, such as CI workflows, lock files or configuration. A change to .github/workflows inside a feature pull request should always be questioned. See the Secure Coding Guide for the underlying practices.

Automated gates before human review

Humans are poor at spotting what is absent and at checking hundreds of lines for patterns. Put the pattern work in the pipeline and make it blocking. On GitHub the relevant ruleset rules are "Require status checks to pass before merging" and "Require code scanning results", the latter blocking merges on alerts at a severity defined in the ruleset. Other platforms have equivalents.

The OpenSSF "Security-Focused Guide for AI Code Assistant Instructions" (August 2025) recommends that the assistant itself be told to generate security tests, including negative tests, and to iterate with scanners such as CodeQL, Bandit, Semgrep or OWASP Dependency-Check until the code passes. That is useful, but treat the assistant's claim that scans pass as unverified: the pipeline result is the evidence. See the CI/CD Security Guide and the Application Security Testing Buyer's Guide.

Using AI to review AI

AI review tools are now common, including Copilot code review and agent-driven review bots. They are fast and catch a share of real issues, but they share the blind spots of the models that wrote the code. GitHub's own documentation says Copilot code review may miss problems, particularly in large or complex changes, can flag problems that do not exist, and should be supplemented with careful human review.

• Use it as triage: let the AI reviewer summarise the change, list touched trust boundaries and flag candidate issues, so the human spends time on judgement rather than reading.
• Never as the approver: an AI review comment must not satisfy a required-approval rule. Keep "Require a pull request before merging" with at least one human approval, and enable "Dismiss stale pull request approvals when new commits are pushed" and "Require approval of the most recent reviewable push" so an agent cannot push after approval.
• Separate the models: where you can, review with a different model or tool from the one that generated the code, and combine it with deterministic scanners whose rules you control.
• Agent-authored pull requests: GitHub rulesets offer "Require an additional approval for unattributed Copilot pull requests" (public preview at the time of checking), which adds a required approval when no person is attributed. Apply the same principle on any platform: a pull request with no accountable human author needs more review, not less.

Provenance and visibility

You cannot sample, route or investigate AI-generated code you cannot identify. Provenance is also what lets you answer an auditor or incident responder asking which changes an agent made and who approved them.

• Commit and pull request markers: require a commit trailer (for example a Co-authored-by line naming the assistant) or a pull request label for AI-assisted changes. Several agents add such trailers by default; make it policy rather than habit.
• Agent identities: agents that open pull requests should do so under their own bot or app identity, not a developer's personal access token. That keeps attribution honest and lets branch rules treat them differently. See the Agentic AI Identity Guide.
• Signed commits: "Require signed commits" ties each commit to a key, which matters when agents can commit locally on a developer's behalf. See the Code Signing Security Guide.
• Telemetry: enterprise assistant plans typically report acceptance and usage per repository. Combine that with markers to estimate the AI share of each codebase and prioritise review capacity.
• Self-disclosure limits: developers forget to label and tools vary. Treat labels as a minimum and assume unlabelled code may also be generated.

Governing the inputs: instruction files and prompts

Instruction files shape what the assistant writes in every session, so they are a high-leverage control and a high-leverage target. The OpenSSF guide recommends concise, specific instructions covering input validation and output encoding, never placing secrets in code, preferring well-known dependencies and verifying that suggested packages exist, secure defaults and least privilege, and generating tests. It also notes that simply telling the assistant to act as a security expert performed worse in some experiments, so persona prompts are not recommended.

• Ownership: add CLAUDE.md, .cursor/rules, .cursorrules, .github/copilot-instructions.md and similar files to CODEOWNERS with the security team as a required reviewer.
• Hidden content: scan these files for zero-width and bidirectional Unicode. In the TrapDoor supply chain campaign, malicious packages wrote .cursorrules and CLAUDE.md files containing invisible instructions, and a linked account opened pull requests proposing similar files to open-source AI projects.
• Baseline content: keep a central, versioned baseline instruction set and let repositories extend it, so security instructions do not drift.
• Not a control on their own: instructions influence output; they do not guarantee it. Every instruction should have a matching pipeline check.

Implementing the review process

1. Inventory usage: find which assistants and agents are in use, in which repositories, and whether any open pull requests under personal tokens. The Shadow AI Discovery Guide covers discovery.
2. Turn on the cheap gates first: secret scanning with push protection and dependency existence checks across all repositories. These catch the highest-impact AI failures with the fewest false positives.
3. Add SAST and IaC scanning in report-only mode: run for two to four weeks on a pilot group of repositories, tune rules, then switch high and critical findings to blocking via "Require code scanning results".
4. Introduce provenance markers: publish the label or trailer convention, add it to instruction files so agents apply it automatically, and add a CI check that warns when an agent identity commits without it.
5. Harden branch rules: human approval required, stale approvals dismissed, most recent push approved, code owner review for workflows, CODEOWNERS and instruction files.
6. Train reviewers: a one-hour session on the flaw table above, with examples from your own codebase, is more effective than generic secure-coding training.
7. Sample and audit: each month, pull a random sample of merged AI-assisted pull requests for a second, security-focused review and feed findings back into rules and instructions.
8. Rollback: any blocking gate should have a documented, time-limited bypass with security approval, and every bypass should be logged. If a gate causes widespread disruption, revert it to report-only rather than removing it.

Decision guidance: matching review depth to risk

Small organisations: rely on platform-native secret scanning, dependency review and a free SAST rule set, plus one strict rule: no self-merging of AI-generated pull requests. Large organisations: centralise gate configuration in organisation-level rulesets, run a package proxy that blocks very new packages, and fund security champions to absorb the review load in high-risk areas.

Detection and measurement

Log and monitor: gate results per pull request, bypasses of blocking gates, merges by agent identities, changes to instruction files and workflows, and new dependencies introduced by AI-assisted pull requests. Useful metrics:

• Escape rate: security defects found after merge in AI-assisted code per 1,000 merged pull requests, compared with non-assisted code.
• Gate catch rate: share of AI-assisted pull requests blocked by at least one security gate, trended over time as instructions improve.
• Provenance coverage: percentage of agent-identity commits carrying the required marker.
• Hallucinated dependency blocks: count of blocked non-existent or very new packages per month.
• Review time on high-risk paths: median human review time for authentication and authorisation changes, to detect rubber-stamping.
• Bypass count: number and age of active gate bypasses.

Common failure patterns

• Trusting a clean diff: approving because the code looks tidy, without checking that authorisation and validation exist.
• Weak review controls on the tools themselves: in the Amazon Q Developer extension compromise, a pull request from an untrusted contributor was merged and shipped a destructive prompt to an AI coding agent in version 1.84.0. Code that controls an agent's behaviour needs the strictest review.
• Letting an AI reviewer count as approval: AI review comments are treated as sign-off, removing the human control the studies show is needed.
• Installing whatever the assistant imports: no existence or age check on new packages, leaving the door open to slopsquatting.
• Tests rewritten to pass: agents weakening assertions to make a failing build green, merged because the pipeline is green.
• Unreviewed instruction files: rules files changed by anyone, including by packages or agents, with no code owner review or Unicode scanning.

Framework mapping

Practitioner checklist

• Enable secret scanning with push protection on every repository where assistants are used.
• Block new dependencies that do not exist in the registry or were published very recently, and require pinned versions.
• Make SAST and IaC scanning required checks, blocking on high and critical findings after a tuning period.
• Require at least one human approval, dismiss stale approvals and require approval of the most recent push; never let an AI review satisfy approval rules.
• Add instruction files, workflows and CODEOWNERS to code owner review and scan them for hidden Unicode.
• Mark AI-assisted and agent-authored changes with a label or commit trailer, and have agents use their own identities.
• Give reviewers the flaw-class checklist and require explicit confirmation of authorisation on new endpoints.
• Review test diffs for weakened assertions and skipped tests.
• Sample merged AI-assisted pull requests monthly for a security-focused second review.
• Track escape rate, gate catch rate, provenance coverage and bypass count.

Standards and references

• NIST: SP 800-218A Secure Software Development Practices for Generative AI and Dual-Use Foundation Models
• OWASP: Top 10 for LLM Applications 2025
• OpenSSF: Security-Focused Guide for AI Code Assistant Instructions
• Pearce et al.: Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions
• Perry et al.: Do Users Write More Insecure Code with AI Assistants?
• Spracklen et al.: We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs
• GitHub Docs: Available rules for rulesets
• GitHub Docs: Responsible use of GitHub Copilot code review

Related NHI Mgmt Group resources: Secure Coding Guide · AI Coding Agents Security Guide · Vibe Coding Security Guide · AI IDE and Coding Agent Security Guide · LLM Security Guide