Join our Newsletter — 33% off our NHI Course

Notifications
Clear all

AI agents in Kubernetes: what changes in production readiness?


(@nhi-mgmt-group)
Member Moderator
Joined: 1 year ago
Posts: 18004
Topic starter  

TL;DR: AI agents deployed to Kubernetes need a production-readiness delta because non-deterministic tool use, runtime identity exercise, and shifting behavior break static workload assumptions, according to ARMO. The core failure is that enforcement gets written before observe and posture produce real runtime evidence, so governance must follow runtime behavior, not manifests.

NHIMG editorial — based on content published by ARMO: Deploying AI Agents to Production Kubernetes: A Security Checklist for Platform Teams

By the numbers:

  • The sensor runs at the kernel layer with single-digit overhead, including 1 to 2.5% CPU and 1% memory, and captures syscall and network activity for every pod in the Deployment.
  • A common pattern is that the agent’s ServiceAccount is granted about 30 to 50 permissions during development and uses only 3 to 5 of them in production.

Questions worth separating out

Q: How should security teams govern AI agents that call internal tools from Kubernetes?

A: They should separate runtime containment from identity governance.

Q: Why do AI agents complicate Kubernetes least privilege?

A: Because their effective privilege is not fixed at deployment.

Q: What breaks when enforcement is applied before observation in agent deployments?

A: Policy gets written against assumptions instead of data.

Practitioner guidance

  • Define a PRR delta for every AI agent deployment Extend the existing Kubernetes production-readiness review with AI-specific checks for runtime identity, dynamic tool use, and behavioral observation before go-live.
  • Capture a runtime AI-BOM before production traffic starts Record the model version, tool endpoints, and transitive dependencies actually loaded at runtime so supply chain review is based on evidence, not manifest intent.
  • Reconcile granted permissions against observed API use Compare the ServiceAccount scope and tool privileges granted at deployment with the APIs and tools the agent actually exercises during the observation window.

What's in the full article

ARMO's full blog covers the operational detail this post intentionally leaves for the source:

  • Implementation specifics for the eBPF observation sensor and the runtime AI-BOM workflow on Kubernetes.
  • Concrete examples of the declared-vs-observed reconciliation report and how the promotion gates are assembled.
  • Per-cloud implementation notes for IRSA on EKS, Workload Identity on GKE, and Defender for Containers on AKS.
  • The step-by-step route from confirmed behavioral patterns to enforceable NetworkPolicy and seccomp profiles.

👉 Read ARMO's checklist for deploying AI agents to production Kubernetes →

AI agents in Kubernetes: what changes in production readiness?

Explore further

View Full Forum →  |  NHI Foundation Course →



   
Quote
(@mr-nhi)
Member Moderator
Joined: 3 months ago
Posts: 17593
 

Runtime behavior, not the manifest, is the real identity boundary for AI agents. The article is right to treat the deployment manifest as an incomplete trust signal because non-deterministic workloads can load different models, tools, and dependencies after startup. In NHI governance terms, that means the identity surface is exercised at runtime, not provisioned once and assumed stable. Platform teams therefore need an operational model that can observe, reconcile, and govern the agent as it behaves, not as it was declared.

A few things that frame the scale:

  • The average estimated time to remediate a leaked secret is 27 days, despite 75% of organisations expressing strong confidence in their secrets management capabilities, according to The State of Secrets in AppSec.
  • Only 44% of developers are reported to follow security best practices for secrets management, which shows how easily runtime access assumptions drift away from actual practice.

A question worth separating out:

Q: Who should approve go-live for AI agents that use dynamic tool calls?

A: Approval should sit with the platform and security owners who can verify the runtime artifacts, not only with application developers. The decision should depend on the observed AI-BOM, declared-vs-observed reconciliation, and confirmed behavioral patterns that show the agent is operating within its intended envelope.

👉 Read our full editorial: AI agent security in Kubernetes needs a production-readiness delta



   
ReplyQuote
Share: