Who is accountable when breaking API changes require updating custom resources before an operator upgrade?

Why This Matters for Security Teams

When an operator upgrade requires custom resources to be updated first, the real risk is not the version bump itself. It is the ownership gap between platform engineering, security operations, and application teams. If schema changes are not validated before rollout, the operator may fail to reconcile, leaving workloads partially managed or exposing stale configuration. NIST guidance on NIST SP 800-53 Rev 5 Security and Privacy Controls treats change control, configuration management, and accountability as core operational controls, not optional hygiene.

This is the same pattern that appears in NHI incidents where teams assume a controller or automated system will absorb schema drift safely. In practice, that assumption breaks when custom resources, API versions, or CRDs change out of sequence. NHIMG research on the Ultimate Guide to NHIs shows how often organisations still lack visibility and disciplined lifecycle control over machine-managed identities and their dependencies. The accountability question matters because the party approving the upgrade is also implicitly accepting the migration risk. In practice, many security teams encounter broken reconciliation only after the upgrade window has already started, rather than through intentional pre-production validation.

How It Works in Practice

Accountability sits with the platform and security operations teams because they control the sequence: inspect the operator release notes, compare old and new CRD schemas, update custom resources, reapply CRDs if required, and verify that the controller can still reconcile objects without manual intervention. That workflow is less about a single technical actor and more about an operational gate. A well-run upgrade process usually includes staging tests, backup of manifests, a rollback plan, and explicit sign-off that every dependent custom resource is compatible.

Practitioners should treat this as configuration governance for a managed control plane. If the operator owns the reconciliation loop, then the operator upgrade is only safe when the desired state it reads has already been migrated. The same principle appears in broader identity and secrets management: Gladinet Hard-Coded Keys RCE Exploitation illustrates how fragile automation becomes when systems depend on assumptions that are no longer valid. For change management, the practical sequence is straightforward:

• Inventory all custom resources and versioned schemas that the operator touches.
• Validate upgrade compatibility in a non-production environment before any production change.
• Update custom resources and CRDs in the order required by the operator documentation.
• Confirm post-upgrade reconciliation, status health, and error-free event logs.
• Assign a named owner for rollback decisions if reconciliation fails.

For teams that manage secrets, credentials, or API-driven automation, this is also where lifecycle discipline matters. NHIMG’s analysis of the McDonald's McHire AI Chatbot Default Credentials incident shows how default or stale configuration can turn automation into an exposure path. These controls tend to break down when operators are upgraded in-place across many clusters with inconsistent CRD ownership because drift makes compatibility impossible to confirm quickly.

Common Variations and Edge Cases

Tighter upgrade controls often increase delivery overhead, requiring organisations to balance release speed against reconciliation safety. That tradeoff becomes sharper in clusters that host multiple teams, where one operator may manage many resource types and not all of them can be migrated on the same schedule. Current guidance suggests treating those environments as higher risk and requiring explicit dependency mapping before any upgrade window.

There is no universal standard for whether application owners or platform teams must perform the resource edits themselves. In most mature operating models, platform engineering executes the upgrade mechanics, while security operations validates policy, auditability, and rollback readiness. The application team may still own the resource content if the schema reflects workload-specific intent. The important point is that accountability cannot be left ambiguous. If a custom resource is generated by CI/CD, the pipeline owner may need to participate in the migration review. If a CRD is deprecated, the owning team should verify whether a conversion webhook or manual rewrite is required. The safe assumption is that operator upgrades are change-management events, not routine package updates.

For regulated environments, best practice is evolving toward documented approval gates, version pinning, and staged rollout sequencing rather than ad hoc remediation after failure. That approach aligns with the broader NHI control gap highlighted by NHIMG, where poor lifecycle control and weak visibility often create avoidable operational exposure.

Related resources from NHI Mgmt Group

• Who is accountable for keeping authorization approvals current when policy changes after a request is submitted?
• Who should be accountable for SAP financial configuration changes that affect general ledger, accounts payable, and integration postings?
• Who is accountable for validating OIDC claims before they are trusted for Kubernetes authorization?
• Who is accountable when shared service accounts or API keys are left exposed?