Notifications
Clear all
Topic starter
02/08/2026 9:45 am
TL;DR: Aggregate scores can hide a material precision gap and a steep drop on large enterprise-style repositories, where Kimi K3 averaged roughly 6% F1 on the biggest codebase tested, according to Semgrep. The result is a reminder that model choice changes triage load, grounded reasoning, and operational risk more than headline benchmarks suggest.
NHIMG editorial — based on content published by Semgrep: Open weight models still lag on enterprise security review tasks
By the numbers:
- Systems with least-privileged AI access had a 17% incident rate vs 76% for over-privileged systems, making over-privileged AI systems 4.5x more likely to experience a security incident.
Questions worth separating out
Q: How should security teams evaluate open weight models for code review work?
A: Start with representative repositories, not toy examples, and measure precision, recall, and F1 against the workflow you actually run.
Q: Why do open weight models behave differently on larger codebases?
A: Large repositories increase the amount of context a model must retain while tracing control flow, dependencies, and data movement.
Q: What do security teams get wrong about AI agent benchmarks?
A: They often measure end-to-end complexity or general model quality instead of the exact failure moment.
Practitioner guidance
- Benchmark on repositories that resemble production Test candidate models against large, interconnected codebases that mirror your own service boundaries, dependency depth, and review patterns.
- Set precision thresholds by analyst capacity Decide how many false positives your team can realistically triage before you integrate the model into a review pipeline.
- Separate data residency from decision governance Treat self-hosting as one control for code locality, not a complete security strategy.
What's in the full article
Semgrep's full post covers the operational detail this post intentionally leaves for the source:
- Per-model benchmark tables for guided prompt and multimodal harness runs across multiple repositories.
- The experiment setup, including the prompts, harness behaviour, and deterministic scoring approach.
- Additional commentary on when open weight models may fit smaller projects versus enterprise-scale repositories.
- The authors' recommendations for balancing precision against triage capacity in security workflows.
👉 Read Semgrep's benchmark analysis of Kimi K3 and open weight security review →
Open weight model benchmarks: are your security controls keeping up?
Explore further
02/08/2026 12:38 pm
Open weight model evaluation is becoming a governance problem, not just a benchmarking problem. Once a model is used to review code, recommend fixes, or influence release decisions, it affects the security control stack rather than sitting outside it. That means organisations need explicit ownership, validation rules, and auditability for the model’s outputs. For identity and security programmes, the practical conclusion is that model-assisted review must be governed like a privileged workflow, not a casual productivity feature.
A question worth separating out:
Q: When should organisations choose a self-hosted model over a frontier model?
A: Choose self-hosting when data residency, local control, or auditability are primary requirements and when you can absorb the extra governance work. But do not assume self-hosting makes a model suitable for every security task. The right decision depends on task fit, performance stability, and the amount of human review still required.
👉 Read our full editorial: Open weight models still lag on enterprise security review tasks
