Open weight models still carry opaque trust risk for AI programs

At a glance

What this is: Semgrep says open weight models still carry opaque trust risk because their behaviour, lineage, and training provenance are hard to verify independently.

Why it matters: IAM, AI security, and governance teams need to treat model provenance and evaluation as control issues, because black box model trust can influence engineering decisions long before defects are obvious.

Context

Open weight does not mean fully inspectable. In practice, AI programs are still being asked to trust model behaviour that cannot be reliably reverse engineered, especially when training data, fine-tuning, or hosting provenance is unclear. That creates a governance gap because the organisation is consuming a decision-making component whose internal logic cannot be validated the way traditional software can.

The identity and supply-chain angle is genuine here: model lineage, ownership of training stages, and the trust placed in third-party model sources all affect security decisions. For AI governance teams, the issue is not only whether a model is accessible, but whether the model's provenance and change history can be independently verified before it is allowed into production.

Key questions

Q: What breaks when model provenance is missing?

A: Without provenance, promotion decisions become difficult to validate and rollback becomes imprecise. Teams cannot confidently prove which code, data and runtime combination produced the model in production, so audits turn into investigation work instead of evidence-based verification. Provenance is what lets governance answer why a model exists in its current state.

Q: Why do benchmarks not prove that a model is safe for production use?

A: Benchmarks only show how a model performed on a test set, not whether its lineage, training data, or post-training changes are trustworthy. A model can be tuned to score well while still carrying hidden behaviours, so production approval needs provenance checks and independent evaluation.

Q: What are the signs that an organisation’s trust model is failing?

A: Common warning signs include relying on static trust assumptions, weak access controls, limited verification of device or user identity, and inconsistent protection across cloud and on-premises environments. Another signal is when teams treat identity checks as a one-time event rather than a continuous control. Those patterns usually show that the trust model is brittle and vulnerable to bypass.

Q: Should organisations require third-party review before deploying external AI models?

A: Yes, for any model that affects code generation, customer outcomes, or regulated decisions. Independent review is the closest analogue to software assurance in traditional supply chains, and it helps validate the model's data sources, training process, and behavioural claims before the organisation depends on it.

Technical breakdown

Why open weight does not equal transparent behaviour

Open weight models expose parameters, but not a readable explanation of how those parameters produce specific outputs. Unlike conventional binaries, where reverse engineering can recover a large part of program behaviour, models remain hard to interpret mechanistically. That means security teams cannot assume that public weights, public code, or visible prompts are enough to establish trust. The practical problem is not just opacity at the interface. It is that behaviour can emerge from training, fine-tuning, and alignment steps that are not obvious from the artefact itself.

Practical implication: Treat model release artefacts as incomplete evidence and require independent evaluation before approving them for business use.

How training provenance creates a hidden trust chain

A model inherits risk from the data and process that produced it. If training data is poisoned, if a post-trained variant is substituted, or if fine-tuning introduces targeted behaviour, the compromise may persist even when the visible model looks normal. This is a supply-chain problem for AI, not just a model-quality issue. The central control question is whether the organisation can trace what was used to build the model and whether that lineage can be reproduced or challenged. Without that, trust becomes a claim rather than a verified property.

Practical implication: Demand lineage evidence for model sources, training inputs, and post-training modifications before model adoption.

Why benchmarks and surface tests are not enough

Benchmarks measure performance against a test set, but they do not prove that a model is trustworthy in operational use. Models can be tuned to pass benchmarks, just as they can be aligned to avoid obvious safety failures while still carrying hidden behaviours. That leaves a gap between observed test results and real-world assurance. For governance teams, this means model approval cannot rest on scores alone. It needs independent review of provenance, reproducibility, and whether the model's behaviour remains stable under the organisation's own use cases.

Practical implication: Use benchmark results as one input, then add independent testing against your own prompts, workloads, and risk scenarios.

Threat narrative

Attacker objective: The objective is to shape model outputs or behaviour in ways that quietly affect engineering, business, or security decisions while avoiding detection.

1. Entry occurs when an organisation adopts an open weight model whose training origin, fine-tuning history, or host provenance cannot be independently verified.
2. Credentialed trust is abused when poisoned data, hidden backdoors, or swapped post-trained variants introduce behaviour that is not visible in the published weights alone.
3. Impact follows when the model generates insecure code, biased recommendations, or hidden trigger responses that influence downstream decisions without obvious signs of compromise.

Breaches seen in the wild

• Ultralytics PyPI compromise 2024: A GitHub Actions injection and a leftover PyPI token let attackers publish four Ultralytics releases that installed a cryptominer.
• Shai Hulud npm malware campaign: Shai Hulud campaign: npm malware exposed secrets on GitHub.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Opaque model trust is becoming a governance problem before it becomes a technical one. Organisations are already making production decisions on the basis of model provenance they cannot fully verify. That shifts the burden from after-the-fact model inspection to pre-adoption assurance, which is much closer to supply-chain governance than traditional application review. Practitioners should treat lineage validation as part of model approval, not as a later audit activity.

Model provenance is the AI equivalent of identity provenance for software supply chains. If the organisation cannot explain where a model came from, what modified it, and which stages were independently checked, trust is only implied. That matters because a model can be harmful without being obviously broken, which makes the control problem about confidence in origin and change history. Security teams should require evidence, not claims, before allowing model reuse.

Benchmark-driven assurance creates a false sense of control. Scores can be gamed, and a model that passes an evaluation suite can still carry hidden behaviours that only emerge in production workflows. That means AI governance needs reproducibility, independent evaluation, and third-party validation as normal control expectations. Practitioners should stop treating benchmark performance as a proxy for trust.

AI model governance now depends on verifiable third-party assurance. Semgrep's argument aligns with a broader security reality: organisations do not earn trust by inspecting only the final artefact. They need outside validation of the development pipeline, training inputs, and post-training modifications. For AI programs, that makes independent evidence the deciding factor, not vendor assurance alone.

Trusted model adoption requires a named concept: provenance trust gap. This is the space between what a team can see in the model artefact and what it can actually verify about lineage, training, and behavioural modification. The gap matters because it is where hidden backdoors and manipulated fine-tuning survive normal reviews. Practitioners should close that gap with explicit provenance controls before scaling model use.

From our research library:

• 92% of organisations expose NHIs to third parties, raising concerns about supply chain security, according to the Ultimate Guide to NHIs.
• Read next: AI Supply Chain Security and AI-BOM Guide

What this signals

Provenance trust gap: The real issue is not whether an open weight model is public, but whether its lineage and modifications can be independently verified before production use. That shifts AI governance toward evidence collection, reproducibility checks, and release gating rather than post-deployment debugging.

AI teams should expect model assurance to look more like supply-chain security than classic model benchmarking. That means stronger review of training inputs, fine-tuning changes, and host provenance, especially where the model influences code, customer decisions, or regulated outputs. According to the Ultimate Guide to NHIs, 92% of organisations expose NHIs to third parties, raising concerns about supply chain security.

For practitioners

• Require model lineage evidence Make provenance a release gate for open weight models, including source, training stages, fine-tuning history, and any post-trained modifications.
• Validate models independently Run your own evaluation set against the model's intended prompts and workflows instead of relying on benchmark claims alone.
• Track model changes as supply chain events Treat fine-tuning, re-hosting, and model swaps as controlled changes that need review, owner sign-off, and traceable records.
• Demand third-party assurance for high-risk use For models that influence code, customer decisions, or regulated outputs, require independent review of the data and training pipeline.

Key takeaways

• Open weight models do not remove trust risk when lineage, fine-tuning, and training provenance remain opaque.
• The central weakness is not visibility into weights alone, but the inability to verify how model behaviour was shaped.
• AI governance should require provenance evidence, reproducible evaluation, and independent review before models are trusted in production.

Key terms

• Model Provenance: Model provenance is the evidence chain showing where an AI artefact came from, how it was modified, and whether the version in use is the one that was approved. For AI security teams, provenance is the control that turns trust from assumption into verification.
• Backdoored model: A backdoored model is an AI model that behaves normally in most situations but produces malicious or undesirable outputs when a trigger condition is met. The trigger may be embedded during training or fine-tuning and can be difficult to detect through ordinary testing. That makes the risk especially relevant for high-trust enterprise workflows.
• Mechanistic interpretability: Mechanistic interpretability is the effort to explain how a model produces its outputs by understanding internal structure and behaviour. It is still a research area, not a dependable operational control. For security teams, its limitation is important because inspectable weights do not automatically reveal hidden triggers or backdoors.
• Independent Model Evaluation: Independent Model Evaluation is the practice of assessing AI systems using external, behaviour-based testing rather than relying on vendor self-attestation. It includes prompt testing, ambiguity analysis, benchmarking, and validation of claims, giving agencies auditable evidence that a model performs as expected under operational conditions.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It suits practitioners who need to connect model trust, workload identity, and governance decisions across AI and security programmes.