AI model supply chain governance now extends into DevSecOps

At a glance

What this is: This is an analysis of how AI model supply chain security is being folded into DevSecOps, with scanning, red teaming, and evidence capture tied to the model lifecycle.

Why it matters: It matters because IAM, security, and compliance teams now need governance for AI models as controlled artifacts, not just code, and that affects access, provenance, review, and auditability across modern identity programmes.

By the numbers:

• 28.65 million new hardcoded secrets were detected in public GitHub commits in 2025 alone, a 34% year-over-year increase and the largest single-year jump ever recorded.
• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.
• 64% of valid secrets leaked in 2022 are still valid and exploitable today, proving that detection alone is not enough without automated revocation.

👉 Read TROJ.AI's analysis of AI model supply chain security in DevSecOps

Context

AI model supply chain security now sits at the intersection of DevSecOps, model governance, and identity control. Once models are versioned, stored, distributed, and embedded into applications or agents, they behave like managed artifacts with security, provenance, and audit requirements.

The governance gap is that many organisations still treat AI as a runtime concern only. In practice, risk enters much earlier, when models are ingested, scanned, tested, approved, and recorded for evidence, which is why model lifecycle controls are becoming part of identity-adjacent security programmes.

Key questions

Q: How should security teams govern AI models in DevSecOps pipelines?

A: Treat models as controlled artifacts with ownership, versioning, provenance, and release approval. Security teams should gate ingestion, scan the artifact, run behavioural tests before deployment, and attach evidence to the model record so audit and release history travel with the version in production.

Q: Why do AI models need more than static scanning before deployment?

A: Static scanning can identify known file and dependency issues, but it does not show how a model behaves under adversarial prompts or multi-turn interactions. AI systems can leak data, produce harmful output, or respond unsafely even when the underlying files look clean, so behavioural testing is essential.

Q: What breaks when model evidence is kept outside the artifact system?

A: Auditability breaks because reviewers cannot reliably prove which version was tested, who approved it, or whether the deployed model matches the tested one. Separate spreadsheets and tickets create gaps in chain of custody, especially when models move quickly through DevSecOps pipelines.

Q: How do teams balance runtime AI monitoring with release-time controls?

A: Use release-time controls to decide what is allowed into production, then use runtime monitoring to detect drift, abuse, or unsafe behaviour after deployment. The two layers solve different problems, and runtime detection cannot compensate for weak pre-deployment governance.

Technical breakdown

Model supply chain security as artifact governance

AI models are increasingly handled like binaries, containers, and packages, which means the security model shifts from one-off validation to artifact governance. A model registry or repository becomes the system of record for versioning, traceability, and provenance. That matters because AI models are not static code alone. They include data, weights, metadata, and behavioural risk that can change between training, storage, and deployment. If those artifacts are not governed at ingestion, the downstream environment inherits unverified trust. Practical implication: treat model registration as a control point, not a storage event.

Practical implication: gate model intake on provenance, ownership, and approval before the artifact reaches build or deployment pipelines.

Automated red teaming for AI models and agents

Automated red teaming tests model behaviour before deployment, using single-turn and multi-turn prompts, algorithmic attacks, and agentic workflows to probe for prompt injection, leakage, toxic output, and other failure modes. This is different from static code scanning because the concern is not just known vulnerabilities in files. It is how the model responds under adversarial interaction. For organisations embedding models into applications and agents, runtime-like behaviour begins early in the lifecycle. The technical value is in repeatability and scale, because manual testing is too slow to keep pace with model churn. Practical implication: build behavioural testing into the pre-deployment path, alongside code and dependency checks.

Practical implication: require repeatable behavioural tests for models that can influence user-facing decisions or agent actions.

Compliance evidence for the AI software supply chain

Evidence capture turns model testing into an auditable control rather than a one-time security activity. When red-teaming results, scan outputs, and approval history are attached back to the artifact record, teams can prove what was tested, when it was tested, and which version was approved. That is the difference between saying a model was reviewed and showing a chain of custody. For regulated environments, that evidence layer becomes part of security accountability, change management, and audit response. Practical implication: preserve test results as immutable artifact evidence, not as separate spreadsheets or ad hoc tickets.

Practical implication: attach test artefacts and approval records to the model itself so audit evidence follows the supply chain.

Threat narrative

Attacker objective: The attacker aims to move harmful model behaviour or embedded risk from a controlled artifact into operational systems where it can cause leakage, misuse, or trust failure.

1. entry: A model enters the organisation through a governed repository or catalog and carries hidden behavioural or file-based risk into the pipeline.
2. credential_harvested: Adversarial prompts, poisoned data, or serialization flaws expose sensitive content, unsafe behaviour, or embedded vulnerabilities during testing.
3. escalation: The risky model is promoted into deployment or embedded into agents, extending the blast radius from one artifact to downstream applications and users.
4. impact: The organisation inherits model leakage, unsafe outputs, compliance gaps, or agent-driven abuse that is harder to detect after release.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• LiteLLM PyPI package breach — LiteLLM PyPI supply chain attack, credentials stolen from users.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI model governance is now a supply chain problem, not a model-only problem. Once models are versioned and distributed like software artifacts, the security boundary moves to the registry, repository, and approval workflow. That means provenance, traceability, and chain of custody are now governance requirements, not optional hygiene. Practitioners should treat model intake as a controlled identity-adjacent event, not a convenience step.

Model red teaming belongs in the build pipeline because behavioural risk exists before deployment. The article describes single-turn, multi-turn, and agentic testing, which is the right frame for modern AI assurance. Static scanning can find file-based issues, but it cannot prove how a model behaves under prompt manipulation or when embedded into an application or agent. Practitioners should assume pre-release testing must include adversarial interaction, not just code analysis.

Audit evidence for AI models is becoming part of security accountability, not just compliance paperwork. Attaching red-team results back into the artifact system creates a record of what was tested and approved. That matters because AI programmes will increasingly need to answer who approved the model, what was checked, and whether the exact version in production matches the version that passed testing. Practitioners should expect auditability to become a deployment gate, not a post-incident exercise.

Runtime AI defence does not replace lifecycle governance, it exposes where lifecycle governance ends. The proposed runtime layer matters because it acknowledges that deployed models continue to behave, drift, and interact after build time. But the deeper lesson is that lifecycle controls and runtime controls serve different purposes. Practitioners need both, and they should not confuse post-deployment monitoring with pre-deployment trust.

AI supply chain security now links developer behaviour, artifact governance, and agent risk into one control plane. Model repositories, build systems, evidence stores, and runtime monitoring are converging around the same trust problem. That convergence means IAM and security teams must think beyond isolated tooling and map ownership, approval, and accountability across the model lifecycle. The practical conclusion is that AI security is becoming a governance discipline, not a point-product checklist.

From our research:

• 64% of valid secrets leaked in 2022 are still valid and exploitable today, proving that detection alone is not enough without automated revocation, according to The State of Secrets Sprawl 2026.
• 28% of secrets incidents now originate outside code repositories, in Slack, Jira, and Confluence, and are 13% more likely to be categorised as critical than code-based leaks.
• To go deeper on lifecycle control, NHI Lifecycle Management Guide explains how provisioning, rotation, and offboarding need to work together.

What this signals

Artifact governance will become a prerequisite for AI assurance. Teams that already separate source, build, and release controls should extend that model to AI models and agents, because versioned artifacts create a trust chain that must be preserved from ingestion to deployment. The signal for practitioners is clear: if the repository cannot prove provenance, the pipeline is not ready for governance.

Model risk will increasingly be judged by evidence quality, not intention. Auditors and internal risk teams will care less about whether a model was tested in principle and more about whether the exact released version carries durable proof of testing and approval. With 64% of valid secrets leaked in 2022 still exploitable today, according to The State of Secrets Sprawl 2026, security teams should assume that delayed or detached evidence has real operational cost.

Behavioural assurance is moving into the same control family as identity and access governance. As AI models gain operational influence, the question is no longer only who can access the artifact but who can approve what it is allowed to do. That aligns AI release governance with broader identity control discipline and makes evidence, ownership, and lifecycle boundaries the deciding factors.

For practitioners

• Classify models as governed artifacts Require provenance, version ownership, and approval status before a model enters build or deployment workflows. Store those attributes with the artifact record so reviewers can trace the exact object that was tested and released.
• Add behavioural tests to pre-deployment gates Run single-turn and multi-turn adversarial tests on models that will influence decisions, customer interactions, or agent actions. Keep those tests repeatable so results can be compared across versions and releases.
• Attach evidence to the artifact itself Preserve scan findings, red-team outputs, and approval records with the model in the repository or catalog. That keeps audit evidence aligned to the released version instead of scattered across tickets and chat logs.
• Separate build-time trust from runtime assurance Use pre-deployment controls to approve what may ship, then use runtime monitoring to detect drift or misuse after release. Do not let post-deployment telemetry substitute for release-time validation.
• Map ownership across AI, DevOps, and compliance teams Define who can register models, who can approve release, and who owns evidence retention. Clear accountability reduces the chance that model risk is accepted by default because no one owns the gate.

Key takeaways

• AI model security is shifting from isolated testing to governed artifact management across the full lifecycle.
• Behavioural red teaming and attached evidence are becoming core controls because static scanning cannot prove how models act under pressure.
• Practitioners should align DevSecOps, compliance, and security ownership around the model record itself, not around separate documentation.

Key terms

• Model Supply Chain: The model supply chain is the set of systems and processes that move an AI model from creation to deployment. It includes registries, build pipelines, approvals, evidence stores, and runtime environments. Security failures anywhere in that chain can expose users to untrusted or unverified model behaviour.
• Behavioural Red Teaming: Behavioural red teaming is adversarial testing that probes how an AI model responds under hostile or unexpected interaction. Instead of checking only code or dependencies, it evaluates prompt injection, leakage, harmful outputs, and agentic misuse before release. For AI governance, this is a control for trust in use, not just trust in code.
• Artifact Evidence: Artifact evidence is the durable record attached to a software or AI asset that proves what was scanned, tested, and approved. In an AI context, it links red-team results, attestations, and release history to the exact version shipped. That makes audit response and accountability materially easier.
• Runtime AI Defence: Runtime AI defence is the monitoring and protection layer that watches an AI system after deployment. It looks for harmful prompts, unsafe outputs, drift, and abuse while the model is in production. It does not replace release-time testing, but it reduces exposure once the system is live.

What's in the full article

TROJ.AI's full post covers the operational detail this analysis intentionally leaves for the source:

• How the TrojAI Detect workflow connects into JFrog Artifactory and Evidence at the artifact level
• What automated red teaming looks like for single-turn, multi-turn, and agentic model tests
• How compliance teams can attach audit-ready evidence to model releases and approvals
• Where the planned runtime defence layer would sit in the broader AI security workflow

👉 The full TROJ.AI post covers the JFrog integration, model scanning, red teaming, and evidence capture flow.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or governance maturity, it is worth exploring.