Topic starter
02/08/2026 9:47 am
TL;DR: Open weight models remain difficult to reverse engineer or reliably assess for backdoors, and their training provenance can be manipulated in ways that affect downstream decisions, according to Semgrep. The practical question is no longer whether models are open, but whether their lineage, evaluation, and trust assumptions are independently verifiable.
Editorial analysis by NHI Mgmt Group, based on content published by Semgrep: “You Can't Reverse Engineer Your Way Out of the AI Supply Chain Problem”.
Key questions
Q: What breaks when model provenance is missing?
A: Without provenance, promotion decisions become difficult to validate and rollback becomes imprecise.
Q: Why do benchmarks not prove that a model is safe for production use?
A: Benchmarks only show how a model performed on a test set, not whether its lineage, training data, or post-training changes are trustworthy.
Q: What are the signs that an organisation’s trust model is failing?
A: Common warning signs include relying on static trust assumptions, weak access controls, limited verification of device or user identity, and inconsistent protection across cloud and on-premises environments.
Practitioner guidance
- Require model lineage evidence Make provenance a release gate for open weight models, including source, training stages, fine-tuning history, and any post-trained modifications.
- Validate models independently Run your own evaluation set against the model's intended prompts and workflows instead of relying on benchmark claims alone.
- Track model changes as supply chain events Treat fine-tuning, re-hosting, and model swaps as controlled changes that need review, owner sign-off, and traceable records.
Bottom line: Open weight models do not remove trust risk when lineage, fine-tuning, and training provenance remain opaque.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
02/08/2026 12:39 pm
Opaque model trust is becoming a governance problem before it becomes a technical one. Organisations are already making production decisions on the basis of model provenance they cannot fully verify. That shifts the burden from after-the-fact model inspection to pre-adoption assurance, which is much closer to supply-chain governance than traditional application review. Practitioners should treat lineage validation as part of model approval, not as a later audit activity.
A few things that frame the scale:
- 92% of organisations expose NHIs to third parties, raising concerns about supply chain security, according to the Ultimate Guide to NHIs.
A question worth separating out:
Q: Should organisations require third-party review before deploying external AI models?
A: Yes, for any model that affects code generation, customer outcomes, or regulated decisions. Independent review is the closest analogue to software assurance in traditional supply chains, and it helps validate the model's data sources, training process, and behavioural claims before the organisation depends on it.
👉 Read our full editorial: Open weight models still carry opaque trust risk for AI programs