AI Infrastructure Security: Workload Identity Guide

Behind every AI agent sits an AI platform: data pipelines, notebooks, training jobs, model registries, inference endpoints, vector databases, AI gateways and GPU clusters, all authenticating to each other with non-human identities that are often over-privileged, long-lived and poorly inventoried. That platform holds some of the most valuable assets an organisation has: training data, model weights, embeddings of internal documents and paid access to expensive models and compute. This guide maps the workload identities in the AI stack, the risks specific to them and the controls for each component. It then covers how to choose identity mechanisms, what to monitor, how the controls align with common frameworks, and a phased roadmap. It is written for platform engineers, MLOps and data engineering teams, cloud security architects and the owners of AI programmes.

Key takeaways

AI platforms create many NHIs. Pipeline and job identities, notebook identities, serving identities, model provider API keys, vector database credentials, gateway keys and storage tokens all need an owner, a scope and a lifetime.
The prizes are data, weights and compute. Stolen AI credentials are used for data theft and for running models at the victim's expense, which Sysdig named "LLMjacking" in 2024.
Use platform-issued, short-lived identities. Kubernetes projected tokens, cloud workload identity and keyless access to model services remove most static keys from notebooks, images and code.
Treat registries and gateways as privileged. Whoever can write to the model registry can replace a model, and an AI gateway holds every provider key behind it.
Notebooks and research clusters are the weak point. They combine human access, broad data permissions, pasted credentials and, too often, unauthenticated network exposure.

The AI infrastructure stack and its identities

Component	Typical identities and credentials	What they can reach
Data ingestion and preparation pipelines	Pipeline service accounts, storage tokens, database credentials	Source systems, data lakes, raw and labelled data
Notebooks and research environments	User credentials, shared service accounts, pasted API keys	Data sets, compute, model providers, often production data
Training and fine-tuning jobs	Job or workload identities, execution roles, registry push rights	Training data, GPU capacity, model registry
Model registry and artefact storage	CI/CD identities, registry tokens, signing keys	Model weights, containers, evaluation data
Inference and model serving	Serving workload identities, model provider API keys	Models, prompts and responses, downstream tools
Vector databases and retrieval	Database API keys, indexing service accounts	Embeddings and chunks of internal documents
AI gateways and model routers	Provider keys for many models, master keys, client credentials	All model providers and tools the organisation uses
GPU clusters and orchestration	Kubernetes service accounts, node identities, scheduler credentials	Compute, mounted data and secrets
Agent runtimes	Agent identities, tool credentials, MCP server tokens	Business systems via tools

Managed ML platforms add their own identities. Amazon SageMaker AI, for example, performs operations "on your behalf" through an IAM execution role; AWS offers the AmazonSageMakerFullAccess managed policy for this and points to more granular roles where a use case requires them. See the Cloud Workload Identity Guide.

Risks specific to AI infrastructure

MITRE ATLAS (release 2026.09) catalogues many of these as adversary techniques, including Valid Accounts (AML.T0012), Unsecured Credentials (AML.T0055), Cost Harvesting (AML.T0034), AI Supply Chain Compromise (AML.T0010) and RAG Credential Harvesting (AML.T0082).

1. Over-broad data access

Training and retrieval pipelines are often given read access to entire data lakes "because the model might need it", so a compromised pipeline identity exposes everything. In the Microsoft SAS Token Exposure 2023, a researcher sharing open-source AI models published a storage URL whose token covered the whole account with full control, and the account held 38TB of private data.

2. Credentials in notebooks, images and data sets

Notebooks mix code, output and credentials, and are shared, exported and committed. Training and serving images often have keys baked in. Data sets can contain secrets too: Truffle Security found 11,908 live secrets in a Common Crawl archive used to train LLMs (12,000 Live Secrets in LLM Training Data 2025). OWASP's LLM02:2025 Sensitive Information Disclosure covers the risk of such data surfacing in model output.

3. Model provider keys and LLMjacking

Model and cloud AI keys are spending credentials. In LLMjacking 2024 to 2026, Sysdig estimated potential victim costs at over $46,000 per day, rising past $100,000 per day with newer models, and a bait AWS key leaked by Permiso was tried within minutes. In the Microsoft Azure OpenAI Abuse 2025, Microsoft says Storm-2139 used leaked customer API keys and resold access. OWASP calls the cost side "Denial of Wallet" under LLM10:2025 Unbounded Consumption. See the LLMjacking Guide.

4. Model and artefact integrity

Whoever can write to the registry can replace a model with a backdoored one. Hugging Face warns that pickle, the default format for PyTorch weights, allows "arbitrary code execution attacks" when loaded, and ATLAS tracks Unsafe AI Artifacts (AML.T0011.000) and Manipulate AI Model (AML.T0018). Dependencies matter as well: the PyTorch torchtriton Supply Chain Attack 2022 stole SSH keys and environment variables from machines installing nightly builds. See the AI Supply Chain and AI-BOM Guide.

5. Vector stores as a data leak path

Vector databases hold embedded chunks of internal documents, often without the original access controls. OWASP LLM08:2025 lists unauthorised access and cross-context leaks in shared vector databases, and recommends "permission-aware vector and embedding stores". ATLAS adds RAG Credential Harvesting, where secrets in ingested documents are retrieved through the model. See the Permission-Aware RAG Guide.

6. Environment mixing

Research, staging and production often share identities and data, so experiments run with production credentials. This is OWASP NHI8:2025 Environment Isolation; SPIFFE likewise advises separate trust domains for environments with different security practices.

7. Exposed clusters, platforms and gateways

Oligo Security says attackers exploited exposed Ray clusters for about seven months in ShadowRay 2024, reaching model provider tokens, cloud credentials and SSH keys (Anyscale disputes that missing authentication is a vulnerability). Wiz found an unauthenticated database holding chat logs and API keys in the DeepSeek Database Exposure 2025. In the LiteLLM MCP Auth Bypass 2026, attackers harvested provider keys from AI gateways, and Wiz found 294 of 3,074 public gateways accepting a default master key.

Controls by component

Pipelines, training and fine-tuning jobs

Run each job under its own workload identity (cloud IAM role, managed identity, Kubernetes service account mapped to a cloud identity, or SPIFFE ID), never under a user account or shared key.
Scope data access to the data sets and prefixes the job needs, read-only unless it writes outputs; replace broad managed policies with granular ones.
Issue credentials for the duration of the job and revoke on completion.
Separate who can start jobs from who can publish models to the production registry.

Notebooks and research environments

Authenticate notebooks with the user's own identity plus a scoped workload identity for compute, not shared service accounts.
Provide secrets through a secrets manager integration, never pasted into cells; scan notebooks and outputs before sharing or committing. See the Secrets Management Guide.
Default research environments to de-identified or sampled data, with approval for production data.
Never expose notebook or cluster dashboards to the internet without authentication; expire idle environments and their credentials.

Model provider access and AI gateways

Route model calls through a gateway that holds provider credentials, so teams authenticate with their own identities instead of holding provider keys.
Use keyless access where offered: Microsoft says Entra ID lets code call Azure OpenAI "without storing an API key", using short-lived tokens for a managed identity granted a role on the resource.
Harden the gateway as a tier-zero system: no default master keys, authentication that fails closed, no direct internet exposure, and rotation of every provider key it holds after any compromise.
Set per-key and per-team spend limits and alerts, and add provider key patterns to secret scanning.

Model registry and artefacts

Treat registry write access as privileged; only controlled pipelines publish to production.
Sign models and verify signatures before loading. The Sigstore model-transparency project, built with the OpenSSF, supports Sigstore, public-key and certificate signing; prefer non-executable formats over pickle.
Record provenance: which data, code and identity produced each model. SLSA's build track describes provenance "showing how the package was built", signed at Build L2.

Inference and serving

Give each serving deployment its own identity with access only to its model and required downstream services.
Authenticate callers to inference endpoints, even internally.
Keep serving identities separate from agent tool credentials. See the MCP Security Guide.

Vector databases and retrieval

Carry source-document permissions into the index and filter retrieval by the requesting user's entitlements.
Partition collections or tenants by sensitivity, and scrub secrets before indexing.
Replace shared database API keys with per-service identities; scope indexing and query rights separately.

GPU clusters

Use projected service account tokens: Kubernetes says they "expire automatically" and the kubelet rotates them, whereas Secret-based tokens "don't expire and don't rotate". Set automountServiceAccountToken: false where a pod needs no API access.
Map Kubernetes service accounts to cloud roles with GKE Workload Identity Federation or EKS Pod Identity, which Google and AWS describe as replacing service account key files and node-wide credentials.
Isolate tenants with namespaces, network policies and separate identities, remembering AWS's warning that containers are not a security boundary. See the Kubernetes NHI Security Guide.

Agent runtimes

Give each agent its own identity, distinct from the serving deployment and from the user it acts for, so its actions can be traced and revoked.
Broker tool credentials at run time with short lifetimes and narrow scopes; never store them in prompts, agent memory or configuration files the model can read.
Treat MCP servers and agent tools as workloads with their own identities and owners. See the Agentic AI Identity Guide.

Choosing the identity mechanism

Where the workload runs	Preferred mechanism	Avoid
Kubernetes (any)	Projected, audience-bound service account tokens via TokenRequest	Long-lived Secret-based tokens
GKE	Workload Identity Federation for GKE	Service account key files
Amazon EKS	EKS Pod Identity or IAM roles for service accounts	Node instance role shared by all pods; static access keys
Azure	Managed identities with Entra ID role assignments	Resource API keys in configuration
Across clouds and on-premises	SPIFFE IDs with X.509-SVIDs (SPIFFE advises X.509 over JWT where possible)	Shared secrets copied between environments
External model providers	Gateway-held keys with per-team identities, or federation where supported	Provider keys in notebooks, images or client code

Small teams on one cloud can rely on that provider's native mechanism and a managed gateway. Organisations training across clouds, on-premises GPU clusters and external providers usually need a common identity layer such as SPIFFE so that policy and audit work the same way everywhere. See the Workload Identity Federation Guide.

Detection and monitoring

Spend and usage anomalies: sudden model invocations, new regions or new models are often the first sign of stolen keys.
Unexpected workloads: new clusters, auto scaling groups or GPU jobs; AWS describes miners running within 10 minutes in the Amazon AWS Crypto-Mining Campaign 2025.
Credential use from outside the platform: a job or serving identity used from an unfamiliar network or interactively.
Registry and index changes: model uploads, tag moves and bulk index exports outside the pipeline.
Secrets in logs: prompt, request and application logs often capture keys; the DeepSeek database held API keys in a log table.
Exposure scans: regular checks for internet-reachable notebooks, dashboards, gateways and vector databases.

Framework mapping

This is an indicative alignment prepared by NHI Mgmt Group, not an official mapping. IDs were checked against the NIST CSF 2.0 text, the NIST SP 800-53 Rev. 5 catalogue, a published ISO/IEC 27001:2022 Annex A list, the OWASP NHI and LLM Top 10 (2025) and MITRE ATLAS 2026.09; CIS Controls v8.1 are given at control level.

Control area	NIST CSF 2.0	NIST SP 800-53 Rev. 5	ISO/IEC 27001:2022 Annex A	CIS v8.1	OWASP / ATLAS
Workload identity per component	PR.AA-01, PR.AA-03	IA-9, IA-5	A.5.16, A.5.17	5	NHI4, NHI7
Least-privilege data access	PR.AA-05	AC-3, AC-6	A.5.15, A.8.3	6	NHI5
Secrets out of notebooks, images and data	PR.DS-01	IA-5(7)	A.5.17	3, 16	NHI2, LLM02, AML.T0055
Model integrity and provenance	ID.RA-09, PR.PS-06	SI-7, SR-4	A.5.21, A.8.25	16	LLM03, LLM04, AML.T0010
Permission-aware retrieval	PR.AA-05	AC-3, AC-4	A.8.3	3, 6	LLM08, AML.T0082
Environment separation	PR.IR-01	SC-7	A.8.31	12	NHI8
Spend and compute monitoring	DE.CM-09	SI-4, AU-6	A.8.16	8	LLM10, AML.T0034

NIST AI 600-1, the Generative AI Profile, frames the same concerns as maintaining "the integrity and (when applicable) the confidentiality of the GAI code, training data, and model weights", and suggests measuring unauthorised access attempts and provenance verification (MS-2.7-004).

Common failure patterns

Over-scoped, long-lived share links: a full-control storage token later extended to 2051, as Wiz reported in the Microsoft SAS case.
Unauthenticated control planes: Ray dashboards, databases and gateways reachable from the internet, as in ShadowRay and DeepSeek.
Default admin keys: AI gateways shipped and left with a known master key.
Keys in shared places: platform-hosted app secrets. Hugging Face revoked tokens and told users to refresh keys after unauthorised access to Spaces secrets (Hugging Face Spaces Breach 2024).
One identity for everything: a single execution role or service account shared by research, training and serving.
Index everything, filter nothing: retrieval that ignores the requester's permissions.
No spend alarms: stolen keys discovered from the invoice.

Metrics that show progress

Metric	What it shows	Target direction
AI workloads using platform-issued, short-lived identities	Static key reduction	Up
Static provider and cloud keys in notebooks, images and repositories	Secret sprawl	Down, towards zero
AI identities with a named owner and documented scope	Accountability	Up, to 100%
Production models with verified signatures and provenance	Artefact integrity	Up
Internet-exposed AI services without authentication	Exposure	Zero
Retrieval indexes enforcing source permissions	Data leak risk	Up
Time from spend anomaly to key revocation	Response speed	Down

Implementation roadmap

Phase	Actions	Outcome
First 30 days	Inventory AI identities and keys; scan notebooks, images and repositories; close internet exposure of clusters, notebooks and gateways; rotate default and leaked keys; set spend alerts	The easiest routes to data and compute are closed
Days 31 to 90	Move jobs and serving to workload identity; centralise provider keys in a hardened gateway; split research, staging and production identities; restrict registry writes	Few static keys, clear separation of duties
Days 91 to 180	Sign and verify models; add provenance; enforce permission-aware retrieval; add detection for unexpected workloads and registry changes; report metrics	Integrity, least privilege and monitoring across the stack

Practitioner checklist

Inventory every identity in the AI stack, with owner, scope and credential lifetime.
Replace static keys with platform-issued, short-lived workload identities.
Scope data access per job and per service; no data-lake-wide read by default.
Keep secrets out of notebooks, images and training data, and scan all three.
Centralise model provider keys in a hardened gateway with per-team identities and spend limits.
Never expose notebooks, clusters, gateways or vector databases without authentication.
Protect registry writes as privileged; sign and verify models; avoid pickle from untrusted sources.
Enforce permission-aware retrieval and scrub secrets before indexing.
Separate research, staging and production identities and data.
Alert on spend anomalies, unexpected workloads and credential use from outside the platform.

Securing AI Infrastructure: Workload Identity for AI Pipelines, Models and GPUs