AI workload discovery in Kubernetes depends on runtime observation

At a glance

What this is: ARMO argues that AI workload discovery in Kubernetes depends on runtime observation because AI agents are invisible to declaration-based inventory methods.

Why it matters: This matters because IAM, NHI, and platform teams cannot govern agent identity, privilege, or access scope until they can enumerate the workloads actually running.

Context

Kubernetes can schedule and run containers, but it does not understand that one container is an AI agent and another is a standard application workload. That creates a governance problem for discovery because the identifying evidence lives in runtime behavior, not in the manifest, tag, or cloud inventory entry. For NHI and agentic AI programmes, the question is not whether AI exists in the cluster, but whether the organisation can prove what it is and who or what it acts as.

The article frames a declaration-observation gap that is broader than AI alone. Declared inventories show what teams said they deployed, while runtime observation shows what actually executed, connected, and consumed external services. That distinction is the difference between a theoretical inventory and a governable control surface, and in Kubernetes it determines whether AI workloads remain shadow AI or become manageable assets.

Key questions

Q: How should security teams find AI workloads that were never tagged in Kubernetes?

A: They should rely on runtime observation, not tags alone. The strongest approach combines process, library, and network telemetry so the platform can infer AI behavior from loaded frameworks, outbound LLM calls, and tool traffic. If discovery depends on developer cooperation, it will miss the workloads most likely to become shadow AI.

Q: Why do declaration-based inventories miss shadow AI in Kubernetes?

A: Because declarations record intent, while AI workload identity is assembled at runtime from prompts, tools, SDKs, and external model calls. If a team relies on CMDBs, tags, or manifests alone, anything deployed without explicit AI labeling remains invisible. That is why shadow AI is a structural inventory gap, not just a governance lapse.

Q: What signs show that an AI workload discovery programme is too weak?

A: If the inventory contains only tagged workloads, if GPU usage is treated as the main detection signal, or if the team cannot explain which workloads are calling LLM endpoints, discovery is incomplete. A weak programme sees declared intent but misses lightweight agents, custom SDK users, and workloads introduced outside normal deployment discipline.

Q: What should teams do after they identify an AI agent in a cluster?

A: They should classify it by role, map it to a service account, and review the privilege attached to that identity before applying policy. An inference server, an orchestrator, and a tool runtime do not need the same access, so governance has to follow the discovered function rather than the container alone.

Technical breakdown

Why Kubernetes metadata cannot identify AI workloads

Kubernetes metadata describes deployment intent, not workload identity. Labels, annotations, image names, and service catalog entries can indicate that a team meant to run an AI workload, but they cannot prove that the running container is actually acting as one. In practice, an AI agent may look identical to any other Python or Node container at the orchestrator layer. The identifying evidence exists inside the process, where loaded libraries, prompt handling, tool use, and external model calls reveal the workload’s real function. Discovery therefore has to infer identity from behavior rather than read it from the platform registry.

Practical implication: Build discovery around runtime evidence, not manifest review or tag hygiene.

The four runtime signal classes that expose AI agents

Effective discovery uses four signal classes. Declared signals are the self-reported labels and annotations. Infrastructure signals are the Kubernetes and node traits, such as GPU requests, model volumes, or accelerated-compute affinity. Library and process signals show what the container actually loaded, including SDKs and agent frameworks. Protocol and network signals show outbound connections to LLM endpoints and MCP-style traffic. The first two classes are useful but incomplete; the latter two are what uncover lightweight agents that never declare themselves and do not need special hardware. A complete inventory depends on all four, because each class covers a different blind spot.

Practical implication: Correlate Kubernetes, runtime, and network telemetry so discovery does not stop at self-declared workloads.

Why observation creates an identity layer for agent governance

Once a workload is discovered through runtime observation, it can be classified by framework, role, and Kubernetes identity. That matters because an inference server, an agent orchestrator, and a tool runtime do not carry the same privilege or risk profile. Discovery becomes the first step in a governable identity model for AI systems, including service account mapping, RBAC review, and later enforcement. Without that runtime-derived identity, least privilege and lifecycle governance remain assumptions rather than controls. In this sense, AI workload discovery is the point where agent identity becomes operational rather than theoretical.

Practical implication: Map discovered AI workloads to service accounts and access scopes before applying policy or certification.

NHI Mgmt Group analysis

Runtime observation is now the control boundary for AI workload governance. In Kubernetes, the inventory problem is no longer about counting containers but about determining which containers are acting as AI systems. The declaration-observation gap means that a manifest-driven inventory can look complete while missing the workloads that matter most. For IAM and NHI teams, the practical conclusion is that agent identity starts at observation time, not approval time.

Shadow AI is an inventory failure, not a policy failure. The article is right to separate the discovery problem from downstream governance because teams cannot govern what they cannot enumerate. This is the same structural lesson seen in NHI programmes: when identity is inferred from runtime behavior, incomplete discovery becomes the root cause of over-permissive access and blind spots in review. Declaration-observation gap: that is the failure mode here, and it is what makes runtime telemetry a prerequisite for policy.

AI workload discovery creates the foundation for least privilege, not the end state. Once workloads are classified, teams can decide whether an agent should have its own service account, whether that account is over-privileged, and whether the workload is a tool-using agent or a simpler inference service. The governance move is to treat discovery as the first identity control, because access reviews and segmentation only work after the workload has been reliably named. Practitioners should treat runtime-derived identity as a control input, not a monitoring nice-to-have.

The operational risk is not just missed visibility, but misclassified trust. A LangChain agent, a vLLM server, and a custom SDK-driven workflow may all appear as containers, yet they demand very different governance. If the classification layer is wrong, privilege, logging, and response controls will all be misapplied. The field needs to stop treating AI discovery as a tagging exercise and start treating it as identity inference under uncertainty.

Agent discovery in Kubernetes is a governance primitive for the broader AI security stack. Discovery determines whether behavioural baselines, authorisation decisions, and audit trails are grounded in reality. That makes runtime observation the enabling layer for AI security programmes that want to move from assumed inventory to verifiable control. The practitioner lesson is clear: if discovery is incomplete, every downstream control inherits that blind spot.

From our research library:

• Only 5.7% of organisations have full visibility into their service accounts, according to the Ultimate Guide to NHIs.
• Read next: AI Agent Identity Security Buyer's Guide

What this signals

Declaration-observation gap: AI security teams should treat inventory as a runtime problem first and a governance problem second. In Kubernetes, the difference between a declared workload and an observed workload is where shadow AI hides, and that gap cannot be closed by better tagging alone.

Runtime-derived identity is the prerequisite for any credible AI access model. If a workload is only known through observation, then service account mapping, RBAC review, and agent classification can finally be aligned to what actually runs, not what someone intended to deploy.

For practitioners

• Implement runtime-based AI workload discovery Use process, library, and network telemetry to identify AI workloads that never appear in tags or CMDB records.
• Correlate declared and observed signals Compare Kubernetes labels, image metadata, GPU requests, outbound LLM calls, and loaded agent frameworks to expose shadow AI.
• Map discovered workloads to service accounts Record the Kubernetes identity, RBAC bindings, and namespace for each discovered AI workload before applying access policy.
• Classify workloads by role and framework Separate inference servers, agent orchestrators, tool runtimes, and custom SDK-driven agents so controls reflect the actual risk profile.
• Test discovery completeness against runtime evidence Run a completeness check that asks whether every active AI workload can be shown through observation, not just declaration.

Key takeaways

• AI workload discovery in Kubernetes fails when teams rely on metadata that cannot express agent behavior, framework use, or outbound model access.
• Runtime observation is the only reliable way to separate declared workloads from shadow AI and build a trustworthy inventory.
• Once AI workloads are discovered, they can be mapped to service accounts and access scopes, which is what makes least-privilege governance possible.

Key terms

• Declaration-Observation Gap: The difference between what teams say is deployed and what is actually running. In AI workload security, the gap widens because runtime behavior often creates the real identity, tool use, and data access profile only after the container starts.
• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
• Runtime Discovery: The process by which an AI agent asks a server what capabilities exist before using them. Unlike static documentation, runtime discovery creates a moving target for security review because new tools can appear without code changes on the client side.
• Workload Classification: The process of grouping workloads by data sensitivity, legal obligation, operational criticality, and recovery requirement. In sovereignty programmes, classification is the decision layer that determines how much control each workload must demonstrate and where those controls must be provable.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security practitioners turn runtime findings into access decisions across identity programmes.