When should organisations continue a deployment after a custom step fails?

When a Failed Step Is a Tolerable Deviation, Not a Release Gate

A custom deployment step can fail without automatically invalidating the release, but only if the step is genuinely non-critical to the control objective it was meant to serve. The practical question is whether the failed action was informational, cosmetic, or redundant, versus whether it was enforcing a safety property, compliance requirement, or trust boundary. If the step was the only place a control existed, continuing means accepting the control gap in production rather than during change management.

For security teams, the key distinction is between a failed convenience step and a failed assurance step. A deployment can sometimes proceed if the environment still has independent checks, such as policy enforcement elsewhere, compensating validation, or post-deploy monitoring that can reliably catch the issue. Where the step contributes to access control, secret handling, approval integrity, or traceability, the safer assumption is that the deployment is no longer equivalent to the tested state. The OWASP Non-Human Identity Top 10 is useful here because many custom steps protect machine credentials, automation trust, or service-to-service access rather than just build hygiene. In practice, many security teams discover that a “non-blocking” failure was actually the only guard preventing a risky release path after the change has already reached production.

What to Check Before Letting the Pipeline Keep Going

The decision should be made against the role of the failed step in the overall deployment chain. If the step validates a dependency, rewrites a configuration, signs an artifact, provisions access, or enforces a policy, then its failure changes the trust properties of the release. If it only generates a report, synchronises a dashboard, or performs a duplicate check already covered by another control, then continuation may be reasonable, provided the exception is visible and reviewed.

• If the step was a gate for safety or compliance, stop the deployment until the control is restored or explicitly waived.
• If the step is duplicated by a stronger downstream control, document the compensating control and verify it is actually active.
• If the failure affects configuration drift, secrets, or identity bindings, treat the deployment as untrusted until those bindings are confirmed.
• If the step only affects observability or reporting, continue only when the missing data will not hide a material change risk.

That judgement becomes especially important when the deployment touches non-human identities, API keys, tokens, or automation accounts, because a seemingly minor failure can leave privileged machine access in an inconsistent state. The control question is not whether the pipeline can technically finish, but whether the release still matches the security assumptions that were approved before the step ran. NIST’s Cybersecurity Framework remains useful as a broad reference point for control integrity and recovery, but the operational decision depends on whether another mechanism will actually catch or contain the specific failure. Where no compensating check exists, continuation breaks the equivalence between tested and deployed state.

Continuing also becomes harder to justify when a failure is silent, because the absence of an error message is not the same as the presence of a validated control. If the pipeline cannot prove the failed step was non-essential, the safer interpretation is that the release condition is unknown rather than acceptable.

Exceptions, Compensating Controls, and the Point Where Continuation Stops Being Safe

Tighter deployment gating often reduces release risk, but it also increases operational friction, so organisations need a clear rule for when a failed custom step is an acceptable exception. The best cases for continuation are narrowly defined: the step is non-security-related, its failure is visible, and a separate control already enforces the same requirement. The moment those conditions weaken, the exception turns into a governance decision rather than an engineering convenience.

There is genuine industry variation on how much failure a delivery process should tolerate. Some teams allow release with a logged exception for non-critical checks; others require every failed custom step to block until manually reviewed. The right answer depends on whether the organisation can prove that the omitted step does not affect access, integrity, compliance evidence, or rollback safety. If it does, then “continue anyway” is not a practical shortcut but a control bypass.

When a deployment continues after a failure, the exception should be time-bound, attributable, and reviewable. The most common mistake is treating “we will watch it later” as a substitute for an actual compensating control. If later detection depends on human attention alone, or if the control failure creates irreversible side effects, continuation is no longer a controlled deviation. In that situation, the failure should be treated as a stop condition until the missing assurance is restored.

Risk and Threat Considerations

The material risk is not the failed step itself, but the possibility that the deployment proceeds with an unenforced control boundary. That can create exposure in access control, change integrity, secret handling, and policy enforcement, especially when the custom step was the only mechanism validating a release assumption.

Failure mechanism: Custom pipeline steps often act as bespoke gates for trust decisions that are not covered elsewhere. When they fail open, are bypassed, or are ignored because the deployment “mostly worked,” the organisation may ship unauthorised configuration, unverified artefacts, or inconsistent identity and permission state into production.

Impact: The result can be an undetected production weakness, a governance gap, or a privileged access path that was never meant to survive release. In the worst case, the organisation loses the ability to prove what was changed, what was authorised, and what controls were actually in force at deployment time.

Practitioner Guidance

Decision rule: Continue only if the failed step is demonstrably non-gating and a separate, working control still enforces the same security or compliance outcome. If you cannot point to that compensating control immediately, treat the deployment as blocked rather than “mostly safe.”

What to verify: Verify what the step actually protected, not what the ticket label says it protected. Custom steps often accumulate hidden authority over approvals, identity bindings, or artefact trust, so the failure assessment should be based on function, not pipeline position.

What practitioners underestimate: Teams often assume later monitoring will catch a bad release, but monitoring is weaker than prevention when the failure changes access, integrity, or rollback state. Once the release has crossed the trust boundary, recovery is usually slower and less certain than stopping early.

Practitioner takeaway: A failed custom step is only tolerable when the organisation can prove the missed control was not the thing preserving trust in the release.

Related resources from NHI Mgmt Group

• How do organisations operationalise NHI ownership at scale?
• When should organisations treat an NHI as a high-priority risk?
• How can organisations reduce the blast radius of compromised agent identities?
• Should organisations prioritise external exposure or internal credential governance first?