They should measure both, but risk avoided is the more meaningful control metric. Output volume can rise even when quality falls, so leaders need evidence that guardrails are preventing insecure patterns, reducing rework, and preserving engineering time.
Measuring AI Coding Risk: Output Volume or Risk Avoided?
Output volume is easy to count, but it is a weak proxy for whether AI-assisted coding is making engineering safer or faster. Risk avoided is the better control metric because it ties measurement to outcomes that matter, fewer insecure patterns, less rework, safer automation, and more reliable developer time. That makes it harder to game and more useful for governance.
Why Output Volume Alone Misleads Teams
Lines generated, tickets closed, or commits merged can all rise while hidden risk rises with them. A team can produce more code and still ship more insecure patterns, more brittle logic, or more review burden. In practice, volume is an activity metric, not a control metric, so it tells you something happened, not whether the system improved.
That distinction matters because AI coding tools often shift effort rather than eliminate it. They can accelerate drafting, but they also introduce verification work, prompt hygiene, dependency checks, and code review overhead. If a measurement system only rewards throughput, it can push teams toward faster production of lower-quality output instead of safer delivery.
Volume also fails to capture whether the AI is constrained well. The useful question is not how much code the tool wrote, but whether the guardrails prevented risky output from reaching production. That includes checking whether the assistant avoided insecure patterns, respected repository boundaries, and reduced the amount of human rework needed to make the code acceptable.
What Risk Avoided Measures Better Than Throughput
Risk avoided focuses on the control effect of the AI system. It asks whether the tooling and workflow prevented bad code, caught unsafe suggestions early, reduced the chance of secret leakage or privilege misuse, and lowered the cost of remediation. That gives leaders a clearer view of whether AI is improving the engineering system rather than merely increasing output.
A practical way to think about it is to measure the downstream burden that AI creates or removes. If the assistant saves time but causes more review escalations, security fixes, and rollback risk, the net benefit is weak. If it produces fewer defects, fewer policy violations, and less manual cleanup, then the risk avoided is real even if raw output looks similar.
This is also where security and productivity align. The control metric should show whether the organisation is preventing avoidable exposure, not just counting artefacts. In a coding context, that usually means tracking whether the tool reduces insecure defaults, avoids unsafe dependency choices, and keeps human reviewers focused on material judgement instead of repetitive correction.
How to Build a Meaningful Measurement Model
Use a balanced scorecard, but let risk avoided carry more weight than volume. A useful measurement set includes secure-output quality, rework rate, review intervention rate, policy violations caught before merge, and time recovered by engineers after filtering out unsafe suggestions. Those signals tell you whether the system is becoming more trustworthy, not just busier.
Leadership should also separate local productivity from enterprise risk. A team may report higher throughput while security and platform teams absorb the cost of reviewing, patching, or reversing the work. If the organisation does not measure those downstream costs, it will systematically overstate the value of the AI tool.
For evidence, compare pre- and post-adoption baselines on defects, remediation effort, and review outcomes. The most persuasive control evidence is not the number of prompts processed, but the degree to which the workflow reduced avoidable risk without shifting the burden elsewhere. That is the metric that supports governance decisions and budget approvals.
Risk and Threat Considerations
AI coding systems can create a false sense of efficiency when they increase output faster than they improve assurance. The main risk is not just bad code, but scaled bad code: insecure patterns, hidden dependency choices, overconfident acceptance of suggestions, and review fatigue that weakens the control environment.
Failure mechanism: Teams optimise for visible volume, so the tool is rewarded for generating more code even when the organisation is paying for that speed through extra review, remediation, or exposure to unsafe changes.
Impact: Leaders get a distorted picture of productivity, while security debt and operational risk accumulate in the background, making the AI programme look successful before its actual control value is proven.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP ASVS, CIS Controls v8 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP ASVS | V15 — Secure Coding and Architecture | AI coding risk is fundamentally about code quality and insecure patterns. |
| Recommendation — Use V15 to judge whether AI-assisted code reduces insecure design and implementation defects. | ||
| CIS Controls v8 | CIS-16 — Application Software Security | The question concerns whether AI code increases or reduces software risk in delivery. |
| Recommendation — Measure whether AI-assisted development lowers software security defects before release. | ||
| NIST CSF 2.0 | PR.DS-10 — Data in Transit is Protected | AI-generated code can affect how data is handled and protected in workflows. |
| Recommendation — Validate that AI-assisted changes preserve required data protection behaviour. | ||
Practitioner Guidance
What to prioritise: Make risk avoided the primary management signal and use volume only as a supporting activity metric. If a dashboard cannot show reduced rework, fewer unsafe suggestions, or lower review burden, it is not measuring control effectiveness.
What to verify: Check that the metric reflects outcomes the business actually cares about, such as fewer insecure patterns reaching review, less time spent correcting AI output, and no increase in hidden operational overhead. If those do not move in the right direction, throughput alone is not evidence of improvement.
Practitioner takeaway: The right question is not whether AI wrote more code, but whether it helped the organisation avoid more risk than it introduced.
Related resources from NHI Mgmt Group
- How can organisations measure whether AI-assisted coding is improving software delivery without increasing risk?
- How should security teams measure the value of AI coding agents instead of tracking completions or usage volume?
- Why do organisations need to measure AI spend by task instead of by token volume alone?
- Why do token-based leaderboards create risk when organisations try to measure AI productivity?