Kubernetes operator scaling exposes secrets management tradeoffs

At a glance

What this is: Infisical argues that a monolithic Kubernetes secrets operator did not scale because per-resource authentication and connection state multiplied overhead.

Why it matters: This matters because IAM and platform teams need secrets architectures that minimize identity duplication, reduce operator churn, and stay manageable as workloads, clusters, and deployment models expand.

👉 Read Infisical's analysis of Kubernetes operator scaling for secrets management

Context

Kubernetes secrets management breaks down when every workload resource carries its own authentication and connection state. The problem is not just performance, it is governance, because duplicated machine identity configuration makes rotation, recovery, and troubleshooting harder as clusters grow.

The article shows a familiar identity lesson in a cloud-native setting: centralization only works if the reconciliation layer does not recreate identity sprawl at the control plane. For teams running NHI programmes, the issue is less about secret delivery mechanics and more about whether the operator reduces or multiplies the number of identity objects you must govern.

Key questions

Q: How should teams design Kubernetes secrets operators for shared machine identities?

A: Use shared connection and authentication objects, not per-resource client state, when many workloads consume the same secret source. That reduces memory use, avoids duplicate authentication traffic during restarts, and makes host or credential changes easier to manage. The goal is to keep the identity layer reusable while each secret resource only declares what it consumes and where it should be written.

Q: Why do per-resource secrets operators break down as Kubernetes clusters grow?

A: They break down because each resource carries its own connection and authentication overhead, which multiplies memory consumption and reconcilation traffic as the cluster scales. When many pods restart together, the operator can create a burst of auth calls and hit rate limits. The deeper issue is the architecture, not just retry logic.

Q: What do security teams get wrong about secrets centralization in Kubernetes?

A: They assume centralization means one store, when in practice the control plane still needs a scalable way to deliver and reconcile secrets into many workloads. If the delivery layer replicates identity state across every resource, centralization becomes an operational burden. Good centralization reduces the number of identity objects that must be governed.

Q: How can platform teams tell whether secret reconciliation is actually healthy?

A: Use readiness status, stable reconciliation timing, and low reauth churn as operational signals. If a restart triggers a wave of repeated authentication attempts or requires mass edits when a host changes, the operator is not healthy at scale. Healthy secret delivery should remain observable and predictable even when workloads roll at the same time.

Technical breakdown

Why per-resource authentication becomes a scaling bottleneck

In the original design, each Kubernetes secret resource stored its own host, authentication, scope, and target reference. That is workable when the number of resources is low, but in Kubernetes each deployment and pod restart can trigger many reconciliations at once. The operator therefore created one client per resource, which duplicated memory use and amplified authentication traffic. In steady state that seems harmless, but under scale it turns configuration convenience into a control-plane liability. Practical implication: treat identity state as shared infrastructure, not as per-resource decoration.

Practical implication: centralize connection and identity state so a restart does not create one auth event per workload resource.

Reference-based CRDs and machine identity reuse

The v2 design splits the configuration into connection, authentication, and secret-sync resources. That separation matters because the controller can cache a client by identity and reuse it across multiple secret resources that share the same auth reference. This reduces duplicate client creation, lowers memory footprint, and avoids mass editing when the host or machine identity changes. It also makes the operator easier to reason about, because the secret resource no longer carries every concern itself. Practical implication: use reference-based resource models when many workloads depend on the same machine identity.

Practical implication: redesign CRDs so one identity object can serve many secrets without replicated client state.

Why readiness and multi-source support matter in secret delivery

The v2 operator adds readiness reporting, multiple source paths, and multiple targets such as Secrets and ConfigMaps. These are not just convenience features. In a large platform, visibility into whether a secret is actually reconciled is part of operational identity governance, and supporting more than one source path reduces the need for separate custom objects. Multi-target writes also acknowledge that not every consumed value has the same sensitivity. Practical implication: secret delivery should expose health, scope, and destination clearly enough to audit at scale.

Practical implication: require readiness status and explicit source-to-target mapping so operators remain observable under scale.

NHI Mgmt Group analysis

Per-resource client replication is the hidden identity tax in Kubernetes secrets operators: When every secret object owns its own authentication and connection state, the control plane inherits the very sprawl secrets management is supposed to remove. The result is not just memory growth, but a governance burden where rotation, host changes, and steady-state reconciliation all multiply by resource count. The practitioner conclusion is that shared identity objects are a structural requirement, not an optimization.

Secrets management platforms should govern machine identity as a reusable reference, not a copied payload: The v2 design reflects a more mature model in which connection and authentication are defined once and consumed many times. That pattern matters because it turns identity from a per-workload burden into a managed dependency. For NHI governance, this is the difference between configuration and control. Practitioners should favor architectures that let one machine identity serve many consumers without duplicating state.

Reference-based reconciliation is a better fit for Kubernetes because the cluster is inherently bursty: Pods restart, deployments roll, and controllers reconcile in waves, so any design that authenticates per resource will eventually fail under synchronized load. The problem is not missing retry logic. The problem is that the resource model itself assumed a slower, more stable runtime than Kubernetes actually provides. The practitioner conclusion is to design for burst tolerance at the identity layer, not just at the application layer.

Identity governance in platform tooling fails when operational convenience becomes a security workaround: The article points to a familiar pattern, where teams tolerate growing friction until workarounds emerge, then the workaround becomes the de facto architecture. That is how secret management ends up with duplicate stores, manual edits, and inconsistent rotation paths. The practitioner conclusion is to remove the friction that encourages shadow configuration before it becomes the control plane standard.

From our research:

• 85% of organisations lack full visibility into third-party vendors connected via OAuth apps, according to The State of Non-Human Identity Security.
• Only 1.5 out of 10 organisations are highly confident in their ability to secure NHIs, compared to nearly 1 in 4 for securing human identities.
• The same pattern appears in Kubernetes operators when shared machine identity is not visible as a governed object, not just a hidden implementation detail.

What this signals

Reference-based identity models are becoming the practical baseline for Kubernetes secret delivery: as clusters grow more dynamic, operators that treat authentication as shared state will be easier to scale, audit, and recover. Teams should expect more secret platforms to converge on this pattern because per-resource identity duplication is operationally expensive.

The broader signal for IAM and platform teams is that control-plane design now matters as much as secret-store design. If the reconciliation layer cannot withstand restarts, scaling, and credential changes without bursts of auth activity, the organisation is carrying identity sprawl into its infrastructure plumbing.

Identity blast radius: the smallest viable governed unit is no longer the workload secret itself, but the reusable identity reference that serves many secrets. That shift will shape how teams think about ownership, rotation, and observability in Kubernetes.

For practitioners

• Refactor shared machine identity state Split connection, authentication, and secret sync into separate reusable objects so one machine identity can serve many workload resources without duplicated clients.
• Limit per-resource authentication ownership Review whether any Kubernetes secret resource still embeds host or credentials data that should live in a shared reference object instead.
• Measure reconciliation bursts after restarts Test operator behaviour during rolling restarts and pod churn to confirm it does not create a spike of simultaneous authentication calls.
• Expose readiness for every secret path Require explicit health reporting for each resource so teams can tell whether a secret has reconciled cleanly or is still retrying.

Key takeaways

• The core problem was architectural, not functional, because per-resource identity state multiplied at cluster scale.
• The evidence was operational, with memory pressure, authentication bursts, and mass-edit overhead showing that the control plane could not absorb growth.
• The fix is to govern machine identity as shared reference state so secrets delivery remains scalable, observable, and easier to rotate.

Key terms

• Reference-based architecture: A reference-based architecture stores shared identity and connection details in reusable objects, then points many workload resources at them. In secrets management, this reduces duplication, lowers control-plane overhead, and makes rotation or host changes easier because the identity state is managed once instead of copied everywhere.
• Machine identity cache: A machine identity cache is a controller-side store of authenticated client state that can be reused across multiple reconciliations. It reduces repeated logins and memory consumption, but it must invalidate cleanly when configuration changes or when authorization fails so stale access does not persist.
• Reconciliation burst: A reconciliation burst is a wave of controller activity triggered when many resources restart, resync, or change at the same time. In Kubernetes, this can produce spikes in authentication calls and memory use, which is why identity-aware controllers need to be designed for burst tolerance, not just steady-state operation.

What's in the full article

Infisical's full blog post covers the implementation detail this post intentionally leaves for the source:

• The exact v1 and v2 CRD structures for InfisicalSecret, InfisicalConnection, InfisicalAuth, and InfisicalStaticSecret.
• The caching and invalidation behaviour that drops a client on config change or a 401 and 403 response.
• The migration path for teams still running v1alpha1 alongside v1beta1.
• The additional support for multi-source paths and multiple targets, including Secret and ConfigMap outputs.

👉 The full Infisical post covers the CRD redesign, cache behaviour, and migration details.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.