AI workload discovery in Kubernetes depends on runtime observation

At a glance

What this is: This is an analysis of why AI workload discovery in Kubernetes depends on runtime observation rather than inventory lookup, and why AI agents are otherwise invisible to normal platform tooling.

Why it matters: It matters because IAM and security teams cannot govern AI agent access, service accounts, or tool use until they can prove which workloads actually exist and what identity they run under.

By the numbers:

• Only 38% have automated certificate lifecycle management in place.
• 57% of organisations lack a complete inventory of their machine identities.

👉 Read ARMO's analysis of how to find every AI agent running in Kubernetes

Context

AI workload discovery is the problem of finding every AI agent, inference server, and tool-using workload that is already running in production. In Kubernetes, that is harder than ordinary asset discovery because the platform does not know what makes a container an AI workload unless runtime evidence reveals it.

The governance gap is the Declaration-Observation Gap: declared inventories say what teams intended to deploy, while runtime telemetry shows what is actually executing. That matters for AI agents because their identity, tooling, and external dependencies often exist only at runtime, which creates a real intersection with NHI governance, service account control, and workload identity.

The article argues that shadow AI is not a tagging problem but an observability problem. That starting position is typical for modern Kubernetes environments and atypical only where organisations already invested in runtime-based discovery and workload classification.

Key questions

Q: How should security teams find AI workloads that were never tagged in Kubernetes?

A: They should rely on runtime observation, not tags alone. The strongest approach combines process, library, and network telemetry so the platform can infer AI behavior from loaded frameworks, outbound LLM calls, and tool traffic. If discovery depends on developer cooperation, it will miss the workloads most likely to become shadow AI.

Q: Why do AI agents on GKE complicate workload identity governance?

A: Because the agent’s cloud permissions, Kubernetes identity, and runtime behaviour all interact with cluster architecture. When platform defaults or project-scoped grants are too broad, the workload may technically be identified but still operate with excessive reach. GKE makes those weaknesses easier to surface, not easier to ignore.

Q: What breaks when discovery only uses cloud asset inventory for AI agents?

A: Cloud asset inventory sees managed services and declared resources, but it misses containers that behave like agents without advertising themselves. The result is incomplete inventory, hidden service account exposure, and access that never enters review. Teams then misjudge their agent population and understate the scope of machine identity governance.

Q: What should teams do when runtime discovery reveals unknown AI workloads?

A: They should pause expansion of those workloads, classify their runtime role, and map the service account or token each one uses before granting broader access. The immediate goal is containment through visibility and identity mapping, because an undiscovered agent cannot be safely authorised until its actual behavior is understood.

Technical breakdown

Why Kubernetes cannot natively identify AI workloads

Kubernetes schedules containers, not AI agents. At the orchestrator layer, a LangChain agent, an inference server, and a web application can all look like the same Python or Node container. The AI character of the workload only appears through runtime behavior such as imported libraries, spawned subprocesses, outbound LLM calls, and tool traffic. That is why lookup fails: there is no native metadata field for agent identity, only indirect evidence that must be observed after the workload starts.

Practical implication: discovery tools must inspect runtime behavior, not just Kubernetes metadata or cloud asset catalogs.

The Declaration-Observation Gap in workload inventory

Every inventory is either declarative or observational. Declarative sources include manifests, labels, asset tags, and CMDB records. Observational sources include process execution, DNS, TLS SNI, and network flow logs. AI workloads widen the gap because developers often do not declare them consistently, and the runtime behavior is assembled from prompts, tools, and model calls that never appear in deployment metadata. Shadow AI is therefore the mismatch between stated state and observed state, not a pure governance failure.

Practical implication: teams should treat runtime telemetry as the source of truth for AI workload inventory.

Four signal classes for AI workload discovery

Effective discovery layers four evidence types. Declared signals catch self-identified workloads through labels and manifests. Infrastructure signals catch GPU-backed or model-heavy deployments. Library and process signals reveal frameworks such as LangChain or OpenAI SDKs loaded into memory. Protocol and network signals catch outbound calls to hosted LLM endpoints or MCP-style traffic. Each class covers a blind spot left by the previous one, which is why no single control can give a complete inventory in a modern cluster.

Practical implication: build discovery as a layered control, then use all four signal classes to classify what each workload actually is.

Threat narrative

Attacker objective: The attacker objective is to conceal AI workloads inside normal infrastructure so they can use hidden runtime identity and access paths without scrutiny.

1. Entry occurs when a developer, operator, or attacker deploys a container that does not declare itself as an AI workload in cluster metadata or asset inventory.
2. Escalation occurs when the workload begins using agent frameworks, model endpoints, or tool servers that only runtime telemetry can observe, while the platform still treats it as generic compute.
3. Impact occurs when undiscovered agents inherit service account access, data paths, and tool permissions without ever being enumerated for governance or review.

NHI Mgmt Group analysis

Runtime observation is now a governance requirement for AI workload security. Declarative inventories are still useful, but they are structurally insufficient for agent discovery because they only reflect what teams say they deployed. AI workload identity emerges at runtime through loaded frameworks, process behavior, and external calls, which means governance has to start with observation. For teams already using NHI and workload identity controls, the lesson is direct: if you cannot enumerate the workload, you cannot govern its service account or privileges.

Shadow AI is an observability gap, not a tagging discipline problem. The article is right to reject the idea that better labels alone will solve discovery. Kubernetes labels, Helm charts, and CMDB records are declarations, and declarations do not capture attacker-introduced workloads or rushed deployments that never made it into governance workflows. The control failure is incomplete visibility, and the practical conclusion is that discovery pipelines need runtime signals before they can support IAM or PAM decisions.

Declaration-Observation Gap: this is the specific failure mode that explains why AI workloads disappear from inventory even when the cluster is healthy and monitored. The gap is between what was approved and what is actually executing, which is why conventional cloud asset management misses agentic systems. In identity terms, this is a lifecycle visibility problem for machine identities that never enter the authoritative record, so practitioners should treat runtime classification as a prerequisite control.

AI workload discovery is becoming the front door to agent governance. Once you can classify an agent, you can map it to a service account, tool surface, and data path, then decide whether its privileges are acceptable. That sequence aligns with NIST CSF and workload identity practices, but the sequencing matters: discovery first, governance second, enforcement third. Practitioners should stop treating discovery as a reporting function and start treating it as the first control in the agent security stack.

AI identities behave more like ephemeral machine identities than like traditional applications. The article shows why agent populations can grow faster than human review cycles, especially when teams rely on runtime-generated behavior and hosted model calls. That creates a direct intersection with NHI governance because the access path is often a service account, token, or tool credential, even when the workload itself is not obvious at deploy time. Security teams should design control points around runtime identity, not just deployment intent.

What this signals

AI workload discovery is becoming part of the identity control plane. Once runtime telemetry reveals which containers are actually acting like agents, teams can stop treating AI discovery as an infrastructure report and start treating it as the intake for service account governance, entitlement review, and workload identity policy. For practitioners, the next step is to connect discovery to identity records so every agent has an owner, an identity, and a revocation path.

The broader programme implication is that observation-based controls will outlast declaration-based controls wherever AI systems are deployed in Kubernetes. That is especially true for hidden or short-lived agents that never appear cleanly in asset tools. Teams should pair cluster visibility with workload identity standards such as the SPIFFE workload identity specification and align governance to runtime facts rather than deployment intent.

For practitioners

• Implement runtime-first discovery Use process, network, and cluster telemetry to identify AI workloads from what they actually load and call, not from tags or CMDB entries alone.
• Map every discovered agent to a workload identity Record the service account, namespace, and tool endpoints for each discovered agent so IAM and NHI controls can be applied to the real runtime identity.
• Classify workloads by observable role Separate inference servers, agent orchestrators, tool runtimes, and RAG components so policy reflects the different access patterns each class creates.
• Validate discovery against hidden-deployment tests Test whether the platform still detects workloads that were not tagged, not catalogued, or deployed outside normal review paths, because that is where shadow AI hides.

Key takeaways

• AI workload discovery fails when teams rely on declared state instead of runtime evidence, because Kubernetes does not natively know what an AI agent is.
• The machine identity problem is already large enough that incomplete inventories are normal, which makes discovery and classification the starting point for governance.
• Security teams should build layered, observation-based discovery so every discovered agent can be mapped to an identity, a role, and an access boundary.

Key terms

• Declaration-Observation Gap: The difference between what teams say is deployed and what is actually running. In AI workload security, the gap widens because runtime behavior often creates the real identity, tool use, and data access profile only after the container starts.
• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
• Runtime Discovery: The process by which an AI agent asks a server what capabilities exist before using them. Unlike static documentation, runtime discovery creates a moving target for security review because new tools can appear without code changes on the client side.
• Workload Identity: The identity assigned to a software workload — such as a containerised application, serverless function, or microservice — enabling it to authenticate to other services without storing static credentials.

What's in the full article

ARMO's full blog covers the operational detail this post intentionally leaves for the source:

• Step-by-step examples of the four signal classes used to identify AI workloads in Kubernetes clusters.
• A practical completeness test that teams can use to measure how much of their AI workload estate is still invisible.
• Examples of how runtime signals separate inference servers, agent orchestrators, and tool runtimes.
• Operational guidance for on-demand workloads that only surface through network telemetry.

👉 ARMO's full post covers runtime signal classes, classification steps, and the discovery completeness test.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management for teams that need to control machine access at scale. It helps practitioners connect discovery, identity, and lifecycle controls into a single operating model.