Topic starter
13/10/2026 1:21 pm
TL;DR: A cheaper open-weight investigation agent outperformed a one-shot frontier model in false-positive triage, cutting input tokens by 97.5%, lowering per-finding cost, and improving accuracy by retrieving repository evidence the larger model could not see, according to Nullify. The real control problem is harness design: allocating model capacity to the stage where evidence can be gathered, not simply maximizing model scale.
Editorial analysis by NHI Mgmt Group, based on content published by Nullify: “Let the Cheap Model Look”.
By the numbers:
- The new agent used 97.5% fewer input tokens.
- Nullify screened 682 findings across six tenants in the first nine days of production.
- The screen caught 99.1% of eventual false positives before the full classifier ran.
Key questions
Q: What breaks when AI triage models cannot inspect the repository context behind a finding?
A: They are forced to decide from partial evidence, which makes benign findings look risky or hides the one fact that proves a finding false.
Q: Why do cheaper models sometimes outperform frontier models in security workflows?
A: Cheaper models can outperform when the harness gives them the right evidence and task boundaries.
Q: How do organisations know if AI triage is actually working?
A: Measure whether the AI improves high-fidelity detection, shortens time to verified response, and preserves reviewer trust in its decisions.
Practitioner guidance
- Define dismissal evidence thresholds Specify what affirmative evidence is required before any AI triage system may dismiss a finding, and force uncertainty into escalation rather than closure.
- Give agents repository retrieval tools Provide read-only tools that let the model inspect callers, constants, config, and related files at the scanned commit so it can verify benignity.
- Separate cost control from decision authority Use cheaper models for bounded evidence gathering, but keep final authority on high-risk dismissals aligned to stricter review when proof is incomplete.
Bottom line: AI security triage improves when the harness can retrieve the evidence needed for dismissal instead of relying on a single large prompt.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
13/10/2026 1:22 pm
AI triage is becoming a governance problem, not just a model-selection problem. Once a security workflow can dismiss findings, the harness becomes part of the control plane. The article shows that the decisive question is not whether a model is frontier-grade, but whether the system can acquire the evidence required for a safe decision. Practitioners should treat AI-enabled triage as an operating control with explicit authority boundaries.
A few things that frame the scale:
- Only 5.7% of organisations have full visibility into their service accounts, according to the Ultimate Guide to NHIs.
- Only 23% of IT leaders were very confident in their organisation's ability to manage security and governance for GenAI deployments, according to a 2025 Gartner survey of 360 IT leaders.
A question worth separating out:
Q: Should organisations trust autonomous dismissal in AI security pipelines?
A: Only when the system can prove benignity with affirmative evidence and has a safe fallback when it cannot. Autonomous dismissal is acceptable for bounded, verifiable checks, but uncertainty should escalate to deeper analysis. That keeps the control asymmetric: false deferral is a cost problem, while false dismissal is a security problem.
👉 Read our full editorial: Cheap models can outperform frontier models in security triage