Join our Newsletter — 33% off our NHI Course

Does splitting AI review into Finder, Critic, and Judge help?

 

(@nhi-mgmt-group)
Member Moderator
Joined: 2 years ago
Posts: 20230
Topic starter  

TL;DR: A 343-rule SAGE pipeline now separates recall, critique, and verdict across 16 vulnerability categories at 1Password, according to 1Password. Security review only becomes governable when the system can challenge its own findings before they drive decisions.

Editorial analysis by NHI Mgmt Group, based on content published by 1Password: “Scaling security reviews at 1Password: Building an AI-powered pipeline”.

By the numbers:

  • SAGE v1 now has 343 rules across 16 vulnerability categories.
  • The average token cost per scan was $0.47 USD in the v0 implementation.

Key questions

Q: Why do AI review pipelines need to treat prompt injection as a security issue?

A: Because the code or context being analysed is untrusted input that can steer the model away from correct judgment.

Q: What are the signs that an AI security review workflow is over-reliant on one model?

A: A strong signal is when discovery, verification, and final approval all come from the same prompt or provider, with no independent challenge stage.

Q: How should security teams use AI-assisted code review safely?

A: Use it as a triage layer that accelerates first-pass detection, then require a separate validation step for findings that affect access control, authentication, secrets, or release gating.

Practitioner guidance

  • Separate finding generation from final adjudication Use different stages, prompts, and evidence inputs for recall, critique, and verdict so the same model is not asked to discover and prove the same issue.
  • Treat prompt injection as a security signal Log injection attempts as their own findings and route them to the same review queue as code defects so model manipulation is observable.
  • Preserve a machine-readable ruleset Keep a compact index of rules and summaries so each finding can be traced back to a specific control, category, and body of guidance.

Bottom line: AI-assisted code review becomes governable only when discovery, challenge, and verdict are separated into distinct stages.

Glossary terms: Progressive Disclosure Pipeline, Prompt Injection (Agentic), Model Agnosticism, False Positive Filter

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →


This topic was modified 1 week ago by NHI Mgmt Group

   
Quote
(@mr-nhi)
Member Moderator
Joined: 6 months ago
Posts: 20712
 

AI-assisted security review is a governance workflow, not a single model prompt. The article shows why discovery, challenge, and verdict need separate control points once a model starts influencing security outcomes. That matters because the assurance question is no longer whether the model can find issues, but whether the surrounding process can prove and explain why a finding stands.

A few things that frame the scale:

  • Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap, according to the State of Secrets in AppSec.

A question worth separating out:

Q: What should teams do when AI review flags look noisy but useful?

A: Preserve the noisy output for recall, then measure which findings survive adversarial critique and final adjudication. The right question is not whether the first pass is perfect, but whether the pipeline can convert broad detection into defensible security decisions.

👉 Read our full editorial: AI code review for security scales by splitting recall from proof



   
ReplyQuote
Share:

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.