Open weight models still lag on enterprise security review tasks

At a glance

What this is: This is Semgrep’s analysis of open weight model performance on IDOR detection, showing that Kimi K3 trails more established models on precision and on large, interconnected repositories.

Why it matters: It matters because security teams evaluating AI-assisted code review need to judge output quality, triage burden, and repository scale together, not assume open weight plus self-hosted equals operationally safe.

By the numbers:

• Kimi K3 averaged roughly 6% F1 on Repo D, the largest enterprise-style repository in our benchmark set.
• Systems with least-privileged AI access had a 17% incident rate vs 76% for over-privileged systems, making over-privileged AI systems 4.5x more likely to experience a security incident.

👉 Read Semgrep's benchmark analysis of Kimi K3 and open weight security review

Context

Open weight models are attractive to security teams because they can be self-hosted, inspected, and tuned for local constraints. The governance problem is that benchmark strength does not automatically translate into reliable security work on real infrastructure, especially when the model is asked to reason across large repositories and produce actionable findings at scale.

The primary concern for IAM and NHI programmes is not only model accuracy. It is how AI-assisted security workflows change who or what can inspect code, propose changes, and influence remediation. Once an AI system is embedded in security review, it becomes part of the control plane and needs tighter governance than a normal productivity tool.

Semgrep’s results suggest that the starting position for many teams is typical: a strong model on paper, a weaker model once the codebase becomes broader, noisier, and more interconnected.

Key questions

Q: How should security teams evaluate open weight models for code review work?

A: Start with representative repositories, not toy examples, and measure precision, recall, and F1 against the workflow you actually run. If the model creates too many false positives or loses grounding on large codebases, it may be cheaper to run than to operate. Include human validation, logging, and repeat tests across multiple harnesses before production use.

Q: Why do open weight models behave differently on larger codebases?

A: Large repositories increase the amount of context a model must retain while tracing control flow, dependencies, and data movement. When that context gets noisy or fragmented, the model can lose grounding, overgeneralise patterns, or invent issues. The failure is often not raw capability but scale-sensitive reasoning collapse.

Q: What do security teams get wrong about AI agent benchmarks?

A: They often measure end-to-end complexity or general model quality instead of the exact failure moment. The result is a score that looks useful but does not isolate whether the backbone, tool access, or orchestration caused the problem. Benchmarks need a specific state, attack vector, and scoring function to be operationally meaningful.

Q: When should organisations choose a self-hosted model over a frontier model?

A: Choose self-hosting when data residency, local control, or auditability are primary requirements and when you can absorb the extra governance work. But do not assume self-hosting makes a model suitable for every security task. The right decision depends on task fit, performance stability, and the amount of human review still required.

Technical breakdown

Why guided prompting changes benchmark results

A guided prompt constrains the model to a single task, defines the vulnerability class, and gives the model a narrow response format. That reduces variance and makes comparison possible, but it also means the score reflects how well the model follows instructions inside a tightly scoped harness rather than how robustly it reasons across unfamiliar code. In security review, that difference matters because real repositories contain messy abstractions, partial context, and patterns that do not map cleanly to a benchmark prompt.

Practical implication: evaluate models in a harness that resembles your real review workflow, not only in isolated benchmark mode.

Precision, recall, and the triage burden in AI-assisted review

Precision measures how many findings are real, recall measures how many real issues the model finds, and F1 blends the two. In a security workflow, low precision shifts cost to analysts because every false positive must be checked and dismissed. That is why a model can look competitive in aggregate while still being operationally expensive. For security teams, the question is not only whether the model finds vulnerabilities, but whether it produces findings at a rate humans can sustainably validate.

Practical implication: set a precision threshold that matches your team’s review capacity before adopting a model in production.

Why large repositories stress grounded reasoning

Large, interconnected codebases increase the chance that a model loses context, overgeneralises patterns, or follows superficial cues instead of tracing control flow and data flow. This is where grounded reasoning becomes the real test: can the model maintain consistency across multiple files, services, and abstractions without inventing issues or missing real ones? The Semgrep results suggest that repository scale can expose weaknesses that do not appear in smaller benchmarks.

Practical implication: benchmark candidate models against repos that match your production scale and architecture complexity.

NHI Mgmt Group analysis

Open weight model evaluation is becoming a governance problem, not just a benchmarking problem. Once a model is used to review code, recommend fixes, or influence release decisions, it affects the security control stack rather than sitting outside it. That means organisations need explicit ownership, validation rules, and auditability for the model’s outputs. For identity and security programmes, the practical conclusion is that model-assisted review must be governed like a privileged workflow, not a casual productivity feature.

Model drift becomes operational debt when security work moves from lab conditions to enterprise repositories. The article shows that aggregate scores can mask a collapse in performance once repository size and interdependence increase. That is a familiar governance failure pattern: controls look acceptable in simplified tests but weaken under production complexity. The named concept here is grounding loss at scale: a model’s ability to stay attached to real code context degrades as the environment becomes larger and more interlinked, which means practitioners should validate performance on representative workloads before relying on it.

Self-hosting reduces data transfer risk, but it does not eliminate model governance risk. The article is right to separate residency concerns from performance concerns. A locally run model may keep code on-premises, yet still generate noisy findings, inconsistent reasoning, or overconfident outputs. Security leaders should therefore distinguish between data control and decision control. The practitioner conclusion is straightforward: self-hosting is a deployment choice, not a governance model.

Security teams should be more concerned with task fit than with benchmark bragging rights. The article’s comparison between GLM, Kimi K3, and frontier models shows that different models can be usable for different workloads. That means the right question is not whether open weight models are “good enough” in the abstract, but where they fit in a controlled workflow, where they do not, and what human review remains mandatory. The conclusion for practitioners is to define task-specific acceptance criteria before adopting any model into review operations.

What this signals

Grounding loss at scale is the operational signal here: models that look usable in small tests can become materially noisier when repository size, branching, and service interdependence increase. Security leaders should therefore tie model adoption to the same sort of control validation they would apply to any privileged automation path, especially where code review affects secrets, access logic, or release gating.

The broader programme implication is that AI-assisted security review needs measurable guardrails, not just model selection. If the tool is being used to find insecure patterns, the surrounding workflow should still enforce review ownership, finding validation, and auditability. That is the same governance logic applied to privileged access: the system may act quickly, but the organisation still needs to know who is accountable when it is wrong.

For identity and secrets programmes, the lesson is consistent with leaked credential governance: detection is not the end state. When AI systems participate in review or remediation, the team also needs controls that limit stale findings, unvalidated actions, and unsupported assumptions. The operational goal is to keep the model inside a bounded decision role, then prove that role holds under realistic workload pressure.

For practitioners

• Benchmark on repositories that resemble production Test candidate models against large, interconnected codebases that mirror your own service boundaries, dependency depth, and review patterns. Small sample repositories can hide grounding failures that appear only at enterprise scale.
• Set precision thresholds by analyst capacity Decide how many false positives your team can realistically triage before you integrate the model into a review pipeline. A model that is cheap to run can still be expensive to operate if it produces too much noise.
• Separate data residency from decision governance Treat self-hosting as one control for code locality, not a complete security strategy. Put approval gates, logging, and audit review around how model findings are accepted, rejected, or escalated.
• Require human validation for high-impact findings Use the model to prioritise review work, but require a human analyst to confirm issues that could affect production access, authentication logic, or release decisions. This limits the risk of overconfident but incorrect recommendations.
• Test model behaviour across multiple harnesses Run the same model through different prompts, plugins, and harnesses to see whether performance is stable or brittle. If results swing widely, the model is not yet dependable enough for security operations.

Key takeaways

• Open weight models can be useful for security review, but benchmark strength alone does not prove production suitability.
• Semgrep’s results show that precision and repository scale matter as much as aggregate F1 when the goal is to keep analyst triage manageable.
• Security teams should govern AI review tools as controlled decision systems, with workload-specific testing, human validation, and clear ownership.

Key terms

• Open-Weight Model: A model whose parameters can be downloaded and run locally by the operator. In practice, that means safety behaviour can be altered outside the provider’s environment, so governance has to focus on the runtime, the operator, and the abuse path, not only the model brand.
• Grounded Reasoning: The ability of a model to keep its outputs anchored to the real code, data, or context it is analysing rather than drifting into plausible but incorrect conclusions. In security work, grounding is what separates useful assistance from noisy or misleading findings.
• Precision: The share of reported findings that are actually real weaknesses. High precision means fewer false positives, but it does not guarantee that the scanner is finding all of the important problems, so it must be read alongside recall to understand practical effectiveness.
• Harness: The harness is the layer of instructions, policies, and approval logic wrapped around an AI agent. It is where organisations try to constrain behaviour, but it only works if the rules are explicit, current, and enforced outside the model itself.

What's in the full article

Semgrep's full post covers the operational detail this post intentionally leaves for the source:

• Per-model benchmark tables for guided prompt and multimodal harness runs across multiple repositories.
• The experiment setup, including the prompts, harness behaviour, and deterministic scoring approach.
• Additional commentary on when open weight models may fit smaller projects versus enterprise-scale repositories.
• The authors' recommendations for balancing precision against triage capacity in security workflows.

👉 Semgrep's full post includes the repo-by-repo results, harness details, and model comparison tables.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity controls to the broader security workflows their programmes depend on.