Should organisations run AI inference in managed services or their own clusters?

If runtime evidence of server behaviour is a security requirement, managed services limit visibility because providers control much of the substrate. Running inference in your own cluster gives security teams access to the serving signals needed for detection and enforcement. The decision should be driven by control requirements, not convenience alone.

Why This Matters for Security Teams

Inference placement is not just a platform choice. It changes who can observe runtime activity, enforce policy, and investigate abuse when an AI system behaves unexpectedly. Managed services can reduce operational burden, but they also narrow the evidence available to the customer. That matters when security teams need to inspect prompts, outputs, tool calls, network paths, or model-serving telemetry to satisfy control objectives. NIST Cybersecurity Framework 2.0 provides a useful baseline for framing the decision around governance, asset visibility, and continuous risk management rather than procurement convenience.

The core issue is control boundaries. If the provider manages the serving layer, the organisation may have limited access to logs, network telemetry, process details, or container-level safeguards. That can weaken detection for prompt injection, model abuse, data exfiltration, and unauthorized tool use. For regulated environments, the question is often whether the provider’s attestation is enough or whether the organisation needs direct operational evidence to prove enforcement. Current guidance suggests that the right answer depends on the sensitivity of the workload, the surrounding data flows, and the evidence needed for audit or incident response. In practice, many security teams discover their visibility gap only after they need to reconstruct an inference-time event, rather than during design.

How It Works in Practice

Organisations that run inference in their own clusters usually gain stronger control over runtime logging, segmentation, secret handling, and policy enforcement. That does not automatically make the environment safer, but it does make it more observable. Security teams can place network controls around the serving layer, inspect container and node telemetry, bind access through least privilege, and integrate the workload into existing monitoring and response pipelines. For AI systems with external tools or agentic workflows, this also makes it easier to apply guardrails around egress, data access, and command execution.

Managed services can still be appropriate when the security model is clear and the provider exposes enough evidence to support governance. Best practice is evolving here, because not every provider exposes the same logs, retention settings, or policy hooks. When comparing options, security teams should ask whether they can validate:

• who can access prompts, outputs, embeddings, and cached artefacts
• whether inference telemetry is available for SIEM and incident response workflows
• how secrets, keys, and service identities are isolated
• what controls exist for network egress, model updates, and version rollback
• how the environment supports detection of prompt injection and data leakage

For a formal risk lens, the NIST AI Risk Management Framework is useful for separating governance and measurement needs from deployment preference, while the MITRE ATLAS knowledge base helps teams map realistic adversarial techniques against AI systems. These controls tend to break down when inference is distributed across ephemeral serverless components because the customer cannot consistently retain or correlate the runtime evidence needed for detection.

Common Variations and Edge Cases

Tighter control often increases cost, operational overhead, and platform responsibility, requiring organisations to balance visibility against speed of delivery. That tradeoff becomes sharper when the model is low risk but the surrounding data is sensitive, or when the team lacks the skill to operate secure GPU or Kubernetes estates. There is no universal standard for this yet, so the most defensible posture is to match deployment model to evidence requirements rather than treat managed services as inherently weak or self-hosting as inherently secure.

Hybrid patterns are common. Some teams keep sensitive inference in their own cluster while using managed services for lower-risk workloads, prototyping, or burst capacity. Others use managed services only when the provider offers strong contractual and technical assurances around logging, isolation, and data handling. Where agentic ai is involved, the identity question also matters: the workload should have a distinct machine identity, scoped permissions, and auditable access to downstream systems. The NIST Cybersecurity Framework 2.0 remains the right umbrella for deciding whether the organisation can detect, respond, and recover with the deployment model it chooses.

In highly regulated environments, managed inference may still be acceptable if the provider can demonstrate control inheritance, retention, and incident support. In shared or multi-tenant settings with limited telemetry, however, the safer default is usually to bring the serving layer into an environment the security team can instrument directly.

Related resources from NHI Mgmt Group

• Should organisations run AI guardrail checks inside their own environment?
• What should organisations check before relying on a managed training platform for custom AI models?
• How should organisations evaluate managed services for data security maturity?
• What should organisations do when third-party AI services change unexpectedly?