Inference is becoming the runtime layer for production AI systems

At a glance

What this is: This is an analysis of why inference is emerging as the runtime layer for production AI systems and how that shifts the infrastructure, cost, and reliability debate.

Why it matters: It matters because IAM, NHI, and platform teams now have to govern model-serving systems, service identities, and tool access as part of production AI operations, not as separate concerns.

By the numbers:

• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.

👉 Read WorkOS's analysis of Fireworks.ai and the inference runtime shift

Context

Inference is the phase where an AI system turns a trained model into a live service, and that service is now what users feel as product speed, reliability, and cost. The primary identity concern is not model training alone, but the runtime estate around it: service accounts, API credentials, routing layers, and the operational controls that let AI features call tools and data sources safely.

That shifts the governance question for NHI and platform teams. Once inference becomes the production runtime, the security boundary moves from a single model endpoint to a distributed serving path that includes secrets, orchestration, telemetry, and deployment controls. For organisations running AI features in production, this is now a workload identity and operational resilience problem as much as an ML problem.

Key questions

Q: How should teams govern identity and access for AI inference platforms?

A: Teams should govern inference platforms like any other production workload with privileged access. That means separating deployment, routing, telemetry, and data-access credentials, scoping each identity tightly, logging every control-plane change, and reviewing who can alter serving behaviour. The key is to treat the runtime path as a governed service estate, not a single API endpoint.

Q: Why do inference stacks create new NHI risk?

A: Inference stacks create new NHI risk because they depend on service identities that can reach model registries, GPU clusters, observability systems, and downstream tools. If those credentials are shared or over-scoped, a compromise or misconfiguration can affect the entire serving path. The risk grows as routing and orchestration become more dynamic.

Q: When does AI serving become a governance problem instead of just an engineering problem?

A: AI serving becomes a governance problem when model choice, routing policy, or fallback behaviour can change production outcomes, costs, or data exposure. At that point, the serving stack is part of control design. Organisations should manage those changes with the same discipline they use for access reviews, change approvals, and production release controls.

Q: What should security teams evaluate before using compound AI systems in production?

A: Security teams should evaluate how the system decides which model to call, what credentials each step uses, and whether fallback paths are visible and auditable. They should also confirm that policy changes cannot silently expand access or change data flow. If the routing layer is opaque, the governance model is incomplete.

Technical breakdown

Inference serving stacks and runtime identity

Inference serving is the production path that receives prompts, loads model weights, manages batching, and returns tokens under latency and cost constraints. In practice, the serving stack includes schedulers, cache management, routing logic, and GPU orchestration, which means the AI runtime is really a set of coordinated services rather than a single model endpoint. That architecture creates identity touchpoints at every layer, from deployment credentials to the APIs that route work across models and clusters. The security issue is not only access to the model, but access to the control plane that decides how inference is executed.

Practical implication: treat inference infrastructure as a governed service estate with scoped credentials, not as a single application API.

Batching, KV cache, and latency trade-offs

Modern inference systems depend heavily on batching and KV cache behaviour to balance throughput against latency. Batching improves utilisation by combining requests, while KV cache reduces repeated computation for long contexts, but both mechanisms create operational complexity when traffic is bursty or request sizes vary sharply. That is why a provider can look fast in one workload and slow in another. The real decision variable is not peak tokens per second, but whether the serving path can hold consistent performance across your actual prompt mix, concurrency shape, and context length.

Practical implication: benchmark your real workloads, including burst patterns and long-context requests, before trusting published performance claims.

Compound AI systems and model routing

Compound AI systems split work across multiple models or steps, such as understanding, generation, verification, and rewrite. This reduces cost and can improve quality, but it also adds routing logic that decides which model gets which task and when fallback occurs. Once those routing decisions are dynamic, the security and governance surface expands beyond a single inference call. Teams need to think about configuration drift, policy consistency, and whether the routing layer can be changed without changing the security behaviour of the whole system.

Practical implication: govern routing rules and fallback paths as production logic, because they shape both reliability and exposure.

Breaches seen in the wild

• MongoBleed breach — MongoBleed exposed secrets across 87K MongoDB servers.
• IOS app secrets leakage report — iOS apps leaking hardcoded secrets and credentials endangering user privacy.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Inference is now the real runtime because product risk has shifted from model choice to service behaviour. The article's central point is that AI features are no longer isolated experiments; they are latency-sensitive product surfaces with operational dependencies. That changes the identity problem from selecting a model to governing the services, credentials, and orchestration that make inference happen. Practitioners should stop treating inference as an AI-only concern and start treating it as a core production workload.

Inference runtime governance is a secrets and service identity problem before it is an AI model problem. Every serving path depends on credentials for deployment, routing, evaluation, telemetry, and downstream tool access. The article makes clear that reliability, cost, and scale come from the stack around the model, which means leaked secrets or over-broad service identities can compromise the entire runtime path. The implication is that AI runtime controls and NHI governance now rise and fall together.

Runtime identity blast radius: This article illustrates how a production AI stack concentrates risk across routing, batching, and model orchestration layers. The model may be the visible asset, but the control plane, model registry, and supporting service identities define the real blast radius when something changes or fails. Practitioners should read this as a warning that the smallest credential or routing change can affect the whole inference path.

Open models do not remove governance pressure, they redistribute it. The story is not that open-weight models simplify security, but that they move differentiation into deployment, evaluation, and operational control. That makes governance more distributed, because the organisation must now control how models are served, tuned, and switched in production. The practitioner conclusion is that AI operating discipline matters more as model access gets easier.

For identity programmes, production AI confirms that runtime controls must extend across human, NHI, and platform layers. Developers may define the desired behaviour, but service identities, access policies, and operational guardrails determine whether the system behaves safely at scale. This is where IAM, NHI governance, and platform engineering converge. The field should expect AI runtime management to become a standard identity governance topic rather than a niche infrastructure concern.

From our research:

• Companies are dedicating an average of 32.4% of their security budgets to secrets management and code security, with US organisations leading at 40.8%, according to The State of Secrets in AppSec.
• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.
• For the runtime identity angle, see Analysis of Claude Code Security for how AI-driven development changes the access problem.

What this signals

Runtime identity is becoming inseparable from AI product design. As inference shifts from a behind-the-scenes service to a customer-facing runtime, organisations need to govern service identities, routing rules, and release controls together. The programme implication is straightforward: if you cannot explain who can change the serving path, you do not yet control the AI runtime.

Inference governance will increasingly sit inside NHI and platform security roadmaps. The operational stack around model serving uses the same kinds of secrets, scoped credentials, and access boundaries that identity teams already manage elsewhere. With 27 days the average time to remediate a leaked secret, according to The State of Secrets in AppSec, the runtime path cannot be left to ad hoc engineering practice.

Model routing is the new policy surface. When systems can dynamically switch between models, the decision logic becomes as important as the model itself. Teams should align that layer with NIST Cybersecurity Framework 2.0 functions for identify, protect, detect, respond, and recover, because AI runtime failures now look operational before they look algorithmic.

For practitioners

• Map every identity in the inference path Inventory the service accounts, API keys, deployment tokens, observability credentials, and cluster permissions that participate in model serving. Remove shared credentials, scope each identity to a single runtime function, and tie access to the serving environment rather than the broader platform.
• Separate serving control from model access Distinguish between who can call the inference API and who can change routing, batching, fallback, or model selection policies. Those controls should sit behind separate approval and logging paths so a routine developer integration cannot alter the runtime behaviour of production AI.
• Benchmark under real traffic shape Test latency, cost, and stability against your actual prompt lengths, concurrency spikes, and model variants instead of relying on vendor averages. Include cold starts, cache effects, and long-context runs in the evaluation so governance decisions are based on the workload you actually run.
• Treat compound systems as policy surfaces If your AI workflow routes across multiple models or steps, define policy for fallback, escalation, and verification as part of change management. A routing rule is not just an optimisation, it is a production control that can expand or shrink the system's effective blast radius.

Key takeaways

• Inference is becoming the operational runtime for production AI, which shifts governance from model selection to service control.
• The surrounding stack, especially service identities, routing logic, and deployment permissions, defines the true blast radius of AI systems.
• Practitioners should benchmark real workloads and govern runtime access together, because cost, latency, and exposure now move as one problem.

Key terms

• Inference runtime: The inference runtime is the live production path that serves model responses to users or systems. It includes the serving API, schedulers, routing logic, caches, deployment credentials, and the infrastructure that keeps the model available under real traffic and latency constraints.
• Compound AI system: A compound AI system is a production workflow that uses more than one model, step, or decision point to complete a task. It may route requests, verify outputs, or rewrite results, which means governance must cover the orchestration logic as well as the underlying model calls.
• Runtime identity: Runtime identity is the set of non-human credentials and permissions used by a live system while it operates. In AI infrastructure, it covers service accounts, API keys, deployment tokens, and control-plane permissions that determine what the inference stack can access or change.
• Model routing policy: Model routing policy is the logic that decides which model, cluster, or workflow step handles a request. It shapes cost, latency, quality, and exposure because it controls when requests are escalated, when fallbacks happen, and which identities are used along the path.

Deepen your knowledge

AI inference governance and runtime identity are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building controls for production AI systems, it is worth exploring.

This post draws on content published by WorkOS: Fireworks.ai: The PyTorch Team's Bet on Inference as the New Runtime. Read the original.