Whenever the workload dynamically loads models, tools, or skills after deployment. In those cases, pre-installation scanning is only a gate for declared inputs, while runtime evidence is the only way to confirm what actually executed and what access paths were opened.
When runtime evidence beats pre-installation scanning
Prioritise runtime evidence when an AI workload can change its effective behaviour after deployment, especially if it dynamically loads models, tools, skills, plugins, or remote dependencies. Pre-installation scanning still helps with declared packages and known inputs, but it cannot prove what executed in production, what identity pathways were used, or what the workload could access once live.
That distinction matters most in AI systems because the runtime surface is often larger than the build-time surface. A clean package manifest does not tell you whether the model fetched a new tool, invoked a hidden connector, or inherited permissions from a surrounding platform.
What pre-installation scanning can and cannot tell you
Pre-installation scanning is strongest when the AI workload has a stable bill of materials: pinned model artifacts, fixed containers, reviewed dependencies, and no runtime extension model. In that case, scanning can reduce obvious supply-chain and configuration risk before anything reaches production.
Its limit is scope. It only validates what is visible before deployment, so it misses post-deploy activation paths, environment-driven configuration, and any runtime decision that depends on live prompts, policy engines, or external services. For AI workloads, that gap is often the difference between a useful gate and a false sense of assurance.
If the workload can pull in code or capabilities after release, the question is not whether the preflight scan passed, but whether the production behaviour remained within the reviewed envelope.
What runtime evidence should prove instead
Runtime evidence should answer the questions that scanning cannot: what actually executed, which tools or services were called, what inputs triggered those calls, and what permissions were exercised in the moment. For AI platforms, this is the only defensible way to confirm the live trust boundary, especially when a model or agent can chain actions across multiple systems.
Good runtime evidence usually includes execution logs, policy decisions, tool invocation records, provenance of loaded assets, and access telemetry that ties action to identity or workload context. When the workload is dynamic, those records are not optional observability extras, they are the control evidence.
For workloads built around workload identity and attested access, runtime evidence should show both the attestation path and the resulting access path. That is where specifications such as the SPIFFE workload identity specification and operational guidance such as Guide to SPIFFE and SPIRE become practically useful, because they help teams connect runtime trust to what the system actually did.
What kinds of AI workloads make runtime evidence non-negotiable
Runtime evidence becomes the primary source of truth when the system can expand at execution time. Common examples include agentic systems that select tools on the fly, AI infrastructure that loads connectors or model variants dynamically, and cloud-hosted workloads that use temporary credentials to reach downstream services. In those settings, the runtime state can diverge materially from the scanned state.
That is also why AI infrastructure identity matters. A pipeline or inference stack may look benign at deployment time, yet still expose high-value access once the job starts. NHIMG’s AI Infrastructure Workload Identity Guide and Cloud Workload Identity Guide both reinforce the same operational point: access evidence at runtime is what tells you whether the workload remained bounded or silently expanded its reach.
Where the AI workload can call tools, the runtime record should also show which tool paths were exercised and whether any unexpected identity or privilege transfer occurred. That is the difference between verifying code quality and verifying operational authority.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Non-Human Identity Top 10 addresses the attack and risk surface, while NIST SP 800-53 Rev 5 sets the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Non-Human Identity Top 10 | NHI-04 — Insecure Authentication | Runtime evidence must confirm the live auth path used by dynamic AI workloads. |
| NHI-08 — Environment Isolation | Dynamic loading can blur reviewed and live environments, making runtime proof essential. | |
| Recommendation — Verify runtime authentication paths and block unreviewed access changes after deployment. Enforce isolation boundaries and validate them with production telemetry. | ||
| NIST SP 800-53 Rev 5 | AU-6 — Audit Record Review, Analysis, and Reporting | Runtime evidence depends on actionable audit records showing what actually executed. |
| SA-11 — Developer Testing and Evaluation | Pre-install scanning is one testing layer, but dynamic AI workloads need runtime evaluation too. | |
| SI-4 — System Monitoring | Live monitoring is needed to see tool calls, loaded components, and access in production. | |
| Recommendation — Review audit records that capture tool use, access paths, and unexpected runtime behavior. Combine predeployment testing with operational checks that validate live behavior. Monitor runtime activity so unexpected execution paths are visible and alertable. | ||
Practitioner Guidance
What to prioritise: If the workload has any post-deployment loading path, prioritise runtime evidence over scan results as the trust anchor. Treat scanning as an upstream filter, not as proof of production behaviour.
What to verify: Confirm that you can reconstruct the live execution path from logs or telemetry, including loaded assets, tool calls, and the access used to make them. If you cannot produce that evidence, the deployment is not fully observable enough for high-trust use.
Decision rule: If the workload can change what it loads or invokes after release, require runtime evidence before you trust its safety or access posture. If the workload is truly static and tightly pinned, pre-installation scanning can carry more of the assurance burden.
Practitioner takeaway: For dynamic AI workloads, the main risk is not that pre-installation scanning is wrong, it is that it is incomplete. Runtime evidence is the only control that can confirm the real execution and access story once the system is live.
Related resources from NHI Mgmt Group
- When should organisations prioritise runtime protection over pre-runtime scanning for cloud workloads?
- When should organisations prioritise runtime attribution over posture scanning for AI agents?
- When should organisations prioritise runtime guardrails over model-focused AI controls?
- When should organisations prioritise runtime AI controls over static approvals?