Cheap models can outperform frontier models in security triage

At a glance

What this is: Nullify argues that AI security triage gets better when cheaper models are given tool access and bounded authority to retrieve evidence before escalating findings.

Why it matters: For security and identity teams, the issue is not just model quality but governance of AI-driven workflows, especially where agents inspect code, make dismissal decisions, and influence remediation queues.

By the numbers:

• The average investigation took 3.9 turns and cost $0.0103 per call.
• The new agent used 97.5% fewer input tokens.
• Nullify screened 682 findings across six tenants in the first nine days of production.
• The screen caught 99.1% of eventual false positives before the full classifier ran.

Context

AI triage systems fail when they are asked to make a final judgment without access to the evidence needed to support it. In security operations, that often means a model is given a local snapshot of a finding but not the repository context, caller chain, or configuration detail that determines whether the issue is real.

This article is about governance of AI security workflows, not just model selection. The identity angle appears where agents are granted read-only authority over repositories and are allowed to make dismissal decisions that shape the security programme's downstream queue and remediation economics.

Key questions

Q: What breaks when AI triage models cannot inspect the repository context behind a finding?

A: They are forced to decide from partial evidence, which makes benign findings look risky or hides the one fact that proves a finding false. In practice, that means the harness must retrieve surrounding code, callers, or configuration before dismissal is allowed. Without that step, you get more false escalations, weaker dismissal accuracy, and unnecessary downstream cost.

Q: Why do cheaper models sometimes outperform frontier models in security workflows?

A: Cheaper models can outperform when the harness gives them the right evidence and task boundaries. A smaller model with tool access can inspect the exact proof it needs, while a larger model without that proof is still guessing. The decisive factor is not raw capability alone but whether the workflow routes evidence to the model at the right moment.

Q: How do organisations know if AI triage is actually working?

A: Measure whether the AI improves high-fidelity detection, shortens time to verified response, and preserves reviewer trust in its decisions. A system that merely closes more alerts is not enough. The right signal is whether the SOC can validate its conclusions quickly and use them in real investigations without rework.

Q: Should organisations trust autonomous dismissal in AI security pipelines?

A: Only when the system can prove benignity with affirmative evidence and has a safe fallback when it cannot. Autonomous dismissal is acceptable for bounded, verifiable checks, but uncertainty should escalate to deeper analysis. That keeps the control asymmetric: false deferral is a cost problem, while false dismissal is a security problem.

Technical breakdown

Why one-shot triage models hit an evidence ceiling

A one-shot classifier can only reason over the context it receives, which makes it brittle when the decisive proof sits outside the local window. In security triage, a function may look unsafe in isolation but become benign once its caller, configuration, or surrounding code is inspected. Bigger models do not solve that if the evidence is still missing. The real limitation is not intelligence alone but evidence acquisition, because a model cannot correctly dismiss what it cannot verify from the available artefacts.

Practical implication: route findings through a context-retrieval step before asking any model to dismiss them.

Tool-using agents change the economics of AI security workflows

A bounded investigation agent can use read-only tools such as Read, Grep, and Glob to pull in the missing evidence instead of carrying huge context upfront. That changes the cost curve because the system spends tokens only on the information needed for the decision. The key architectural shift is not that the model is smaller, but that the harness lets the model gather proof. This is a workflow design issue as much as a model choice issue, and it becomes central when AI work runs continuously.

Practical implication: design AI triage harnesses around evidence retrieval, not around maximum prompt size.

Why dismissal authority must be asymmetric

If an AI screen is allowed to dismiss a finding, the failure model matters more than the average case. A wrong dismissal suppresses a real issue, while a wrong deferral only adds downstream cost. That is why the article's architecture requires affirmative evidence before dismissal and sends uncertainty to the deeper triage path. This is a governance pattern for AI systems: grant narrow authority, define the stopping condition tightly, and make uncertainty escalate rather than silently conclude.

Practical implication: require explicit proof for autonomous dismissal and treat uncertainty as an escalation trigger.

Threat narrative

Attacker objective: The practical objective is not theft but false assurance: to let an unresolved issue escape scrutiny or to force excessive compute and analyst effort on harmless noise.

1. Entry occurs when scanner output feeds the AI triage pipeline with a finding that appears suspicious in local context but is not yet proven benign.
2. Credential access is not the issue here; the crucial step is evidence access, where the agent retrieves repository context through Read, Grep, and Glob to test the finding against surrounding code.
3. Impact occurs when the harness wrongly dismisses a real issue or wastes budget by over-analysing noise, so the security programme depends on correct evidence gathering and escalation.

Breaches seen in the wild

• 12,000 secrets in LLM training data: Truffle Security found 11,908 live API keys and passwords hard-coded in web pages captured by Common Crawl, a dataset used to train LLMs.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI triage is becoming a governance problem, not just a model-selection problem. Once a security workflow can dismiss findings, the harness becomes part of the control plane. The article shows that the decisive question is not whether a model is frontier-grade, but whether the system can acquire the evidence required for a safe decision. Practitioners should treat AI-enabled triage as an operating control with explicit authority boundaries.

Evidence retrieval is the real differentiator in security automation. In this kind of workflow, the best result comes from giving a cheaper model the ability to inspect the repository, not from asking a stronger model to guess with incomplete context. That changes how teams should think about AI cost, because the model tier matters less than the evidence path. The practitioner conclusion is to optimise the harness before chasing larger models.

Agentic security systems create a new trust boundary around infrastructure quality. The article's note about incomplete repository clones shows that checkout fidelity, tool permissions, and auditability are now part of the security boundary. That is the identity bridge: when an AI agent is authorised to inspect code and decide on dismissal, its authority resembles a tightly scoped non-human identity. Practitioners need governance over the agent's access, not just over its prompt.

Cheap model capacity will increasingly outrank raw frontier capacity for bounded security tasks. The market signal is that continuous AI operations will reward architectures that allocate expensive reasoning only where it adds real value. This does not remove the need for frontier models, but it does force teams to justify where they are consuming them. The practitioner conclusion is to build allocation logic, not one-size-fits-all model policy.

Security programmes need measurable evidence thresholds for autonomous action. The article's asymmetric failure model is the right pattern for AI security operations: prove benignity before dismissal, escalate uncertainty, and keep the deeper pipeline intact. That approach aligns with NIST AI RMF governance logic and with broader AI operational controls. Practitioners should document what evidence is sufficient for AI-driven dismissal and what must always escalate.

From our research library:

• Only 5.7% of organisations have full visibility into their service accounts, according to the Ultimate Guide to NHIs.
• Only 23% of IT leaders were very confident in their organisation's ability to manage security and governance for GenAI deployments, according to a 2025 Gartner survey of 360 IT leaders.
• Read next: AI Agent Identity Security Buyer's Guide

What this signals

AI triage economics are shifting toward evidence retrieval rather than model scale. As models get cheaper and more capable, the winning design is increasingly the one that can gather the right proof before deciding. That is a governance change for security programmes: model selection matters, but harness design now matters more.

Repository-context access is becoming a control boundary for agentic security systems. If an investigation agent can inspect callers, constants, and configuration, then its authority resembles a narrowly scoped non-human identity and must be governed that way. Teams should treat checkout fidelity, read-only permissions, and audit trails as part of the control set.

5.7% full visibility into service accounts remains a useful cautionary benchmark for identity teams, according to the Ultimate Guide to NHIs. Even in AI-heavy workflows, poor discovery and incomplete inventory undermine confidence in what a system can safely decide. The practical lesson is that visibility and evidence quality are the same governance problem in different forms.

For practitioners

• Define dismissal evidence thresholds Specify what affirmative evidence is required before any AI triage system may dismiss a finding, and force uncertainty into escalation rather than closure.
• Give agents repository retrieval tools Provide read-only tools that let the model inspect callers, constants, config, and related files at the scanned commit so it can verify benignity.
• Separate cost control from decision authority Use cheaper models for bounded evidence gathering, but keep final authority on high-risk dismissals aligned to stricter review when proof is incomplete.
• Audit repository fidelity and tool failures Treat incomplete clones, checkout defects, and tool errors as control issues because they can change what the agent concludes about a finding.
• Measure security outcome per token spent Track how much downstream analysis each screening stage removes, then reassign frontier capacity only to work that truly needs deeper reasoning.

Key takeaways

• AI security triage improves when the harness can retrieve the evidence needed for dismissal instead of relying on a single large prompt.
• The article shows that cheaper models can be more accurate when they are allowed to inspect the repository context behind a finding.
• Security teams need explicit authority boundaries for AI agents because dismissal decisions are now part of the control plane.

Key terms

• AI triage harness: An AI triage harness is the surrounding workflow that decides what context a model sees, which tools it can use, when it may stop, and when uncertain work escalates. In security programmes, the harness often determines whether AI output is safe enough to act on, not just whether the model is smart enough.
• Evidence retrieval: Evidence retrieval is the process of fetching the specific artefacts a model needs to make a grounded decision, such as callers, constants, configuration, or related files. In agentic security workflows, it replaces guesswork with proof and often matters more than larger context windows.
• Asymmetric failure model: An asymmetric failure model gives different consequences to different mistakes. In this article's context, a wrong dismissal is treated as dangerous because it can suppress a real issue, while a wrong deferral is merely expensive because it sends work downstream for deeper analysis.
• Tool-using agent: A tool-using agent is a software system that can call controlled functions to gather data or inspect systems before it acts. In security triage, the agent's value comes from being able to inspect repository state and retrieve evidence, not from producing a single static answer.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and agentic AI identity. It is a fit for practitioners who need to align identity controls with emerging AI operational models.