What do security teams get wrong about signed models and agent trust?

They often assume that authenticating the agent is enough. In reality, identity assurance and artefact assurance are separate problems: a legitimate agent can still run a poisoned or substituted model. Signed models close that gap by proving the execution artefact matches a trusted provenance chain before the agent is allowed to use it.

Why This Matters for Security Teams

Signed models are often treated as a box-ticking control for supply chain trust, but that mindset misses the real risk: the agent may be authenticated while the model it loads is not the one security approved. For autonomous workloads, identity assurance and artefact assurance are separate problems. A valid principal can still execute a poisoned, swapped, or downgraded model if provenance is not enforced at runtime.

This is why model signing belongs in the same conversation as workload identity, policy enforcement, and software supply chain verification. Guidance from the NIST AI Risk Management Framework and the OWASP Agentic AI Top 10 both point to the same operational issue: trust decisions must extend beyond who is calling the model to what artefact is actually being executed. NHI Mgmt Group sees the same pattern across agentic environments, where teams harden access to the agent but leave the model supply chain under-governed, creating a blind spot similar to the exposure described in the Ultimate Guide to NHIs. In practice, many security teams discover this only after a signed agent has already loaded an untrusted model version, rather than through deliberate provenance checks.

How It Works in Practice

The practical control is to verify the model artefact before execution, not after the agent has already started using it. A signature should bind the model file, version, hash, and provenance metadata to a trusted signer, then the runtime should reject anything that does not match policy. This is closer to software attestation than to traditional IAM, because the decision is about artefact integrity, not user intent.

For agentic systems, best practice is evolving toward layered verification:

• Workload identity establishes what the agent is through cryptographic identity, such as SPIFFE or OIDC-backed workload credentials.
• Policy checks determine whether that agent is allowed to load a specific model class, source, or version at request time.
• Signature validation confirms the artefact has not been substituted, tampered with, or rolled back.
• Ephemeral access limits reduce the blast radius if a model or signing key is compromised.

That layered model aligns with current guidance in the CSA MAESTRO agentic AI threat modeling framework and the OWASP NHI Top 10, both of which emphasise that agent trust depends on runtime governance, not static approval alone. Where teams often go wrong is assuming a trusted registry is enough, when the real control is end-to-end provenance enforcement from build to deployment to invocation. These controls tend to break down in loosely managed multi-model environments because model selection happens dynamically and policy coverage becomes inconsistent across pipelines, edge nodes, and third-party hosting.

Common Variations and Edge Cases

Tighter artefact verification often increases operational overhead, requiring organisations to balance stronger trust guarantees against release speed and model agility. That tradeoff becomes sharper when teams are experimenting with frequent retraining, blue-green model swaps, or external model marketplaces.

There is no universal standard for this yet, so current guidance suggests treating signed models as one control layer rather than a complete trust model. Some environments only need signature checks on high-risk models, while others require mandatory verification for every production load. The right answer depends on how much autonomy the agent has, whether the model can trigger tool use, and how quickly the environment can revoke trust if a signer or registry is compromised.

Teams also miss edge cases where the signature is valid but the model is still unsafe. A signed malicious model, a trusted model with harmful prompt behavior, or a legitimate rollback to an older vulnerable version can all pass a naive integrity check. That is why provenance review should be paired with runtime monitoring, approved source enforcement, and revocation workflows. The risks become more visible in incidents like the Analysis of Claude Code Security and the Amazon Q AI Coding Agent Compromised, where trust in the agent was not enough to prevent harmful execution paths.

Related resources from NHI Mgmt Group

• What do security teams get wrong about trust in zero-trust access models?
• What do security teams get wrong about Zero Trust and identity governance?
• What do security teams get wrong about local-only agent controls?
• What do security teams get wrong about AI agent identity governance?