Join our Newsletter — 33% off our NHI Course
Architecture & Implementation

Model Serving

← Back to Glossary
By NHI Mgmt Group Updated November 2, 2026 Domain: Architecture & Implementation

The production phase where a trained model is exposed through an application or endpoint so it can respond to requests. In identity terms, serving introduces a distinct trust boundary because the runtime may access specialised compute, model artefacts, and upstream systems.

What Model Serving Means in Practice

Model serving is the production stage where a trained model becomes an active system, exposed through an endpoint or application so it can answer requests. That shift matters because the model is no longer just an artefact to evaluate, it is now a runtime dependency with availability, access, and trust requirements.

For practitioners, the key point is that serving turns model outputs into an operational service. The quality of the model still matters, but the service boundary now also includes request handling, authentication, latency, scaling, logging, and the systems the model can reach.

Why the Serving Boundary Changes Security Thinking

Once a model is served, the attack surface expands beyond the model file itself. The endpoint can be probed, overloaded, misused, or abused as a bridge into repositories, feature stores, vector stores, or other internal systems that the runtime can access.

This is why model serving is usually treated as a separate control plane from training. A model that is safe in a notebook or offline evaluation can still create exposure when it is reachable by users, applications, or other services. The security question becomes not only “is the model accurate?” but also “who can call it, what can it see, and what can it trigger?”

Operational Characteristics of Serving

Serving usually involves an inference server, gateway, or application layer that accepts prompts, payloads, or API calls and returns predictions. In mature environments, that layer may also manage batching, caching, routing, model version selection, and telemetry.

Those operational choices affect both reliability and risk. For example, shared infrastructure can improve efficiency, but it also concentrates failure impact. Low-latency designs may reduce inspection opportunities, while highly distributed serving paths can make inventory and policy enforcement harder.

Serving also creates a lifecycle issue: models change, versions shift, endpoints are rotated, and permissions drift. A served model is therefore not a static artefact but a managed production asset that needs ownership and change control.

Control Objectives for Secure Serving

The main control objectives are to limit who can reach the service, restrict what the runtime can access, and preserve visibility into requests and responses. That includes endpoint authentication, least-privilege permissions for any upstream systems, and clear separation between the model runtime and sensitive data or administrative functions.

For AI infrastructure and serving-specific hardening, see AI Infrastructure Workload Identity Guide, which covers the identities behind AI platforms, inference, and GPU clusters. For vendor-specific serving platforms, vLLM, Triton and NVIDIA NIM Inference Server Security Guide explains gateway authentication, control endpoints, model repositories, and logging. Databricks Mosaic AI and Agent Bricks Security is useful where serving is embedded in a broader platform with authorization, secrets, and egress controls.

Risk and Threat Considerations

Model serving concentrates risk because a live endpoint can be attacked directly, misused through automation, or used as a stepping stone into connected systems. The most material failure modes are unauthorised access, weak segregation between tenants or environments, and exposed runtime credentials or upstream connections.

Failure mechanism: Attackers or unauthorized users abuse the serving endpoint, exhaust resources, or exploit the runtime’s network and permissions to reach data, tools, or internal services that the model should not access.

Impact: The result can include service disruption, data exposure, unauthorised model use, leaked secrets, or a broader compromise path through the serving environment.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

NIST SP 800-53 Rev 5 provides the primary governance reference for this term.

FrameworkControl / ReferenceRelevance
NIST SP 800-53 Rev 5IA-2 — Identification and Authentication (Organizational Users)Served model endpoints need authenticated access for human operators and users.
IA-9 — Identification and Authentication (Non-Organizational Users)Serving often exposes APIs or endpoints to external callers and client applications.
AC-6 — Least PrivilegeServing runtimes often access data, tools, or internal services and need constrained permissions.
Recommendation — Require authenticated access to serving endpoints for human users and administrators. Authenticate external callers before allowing them to invoke the model service. Constrain the serving runtime to the minimum access needed for inference.

Practitioner Guidance

Why practitioners should care: Treat serving as a production trust boundary, not just a deployment detail. The serving layer is where model access, application access, and infrastructure access collide, so ownership needs to be explicit.

What to watch for: Pay close attention to unauthenticated endpoints, broad network reachability, reused credentials across environments, and gaps between the model runtime and the systems it can query. Those are the conditions that most often turn a useful inference service into a security liability.

Practitioner takeaway: If the model can respond to requests, it should be governed like any other externally reachable production service, with clear access control, telemetry, and change management.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Reviewed and updated by the NHIMG editorial team on November 2, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org