CI/CD server rollout playbook: phased implementation with security built in

A new CI/CD server becomes the most privileged non-human identity in the estate within weeks of go-live, and most of the weaknesses it will carry for years are decided during rollout. This playbook sequences the implementation of a continuous integration and delivery platform, self-hosted or delivered as a service, so that administration, credentials, build agents, pipeline governance and logging are designed in rather than retrofitted. It is vendor-neutral and written for platform engineering, DevSecOps and security architecture teams planning a first platform or a second, consolidated one. Control depth lives in the CI/CD Security Guide and the CI/CD Pipeline Identity Security Guide; this page is about order, gates and evidence, and links out rather than repeating them. If you are moving pipelines off an existing platform, use the CI/CD server migration playbook alongside this one.

Key takeaways

• Design identity before the first pipeline. Decide how people, agents and deployment jobs authenticate before any team onboards, because credentials created in week one become the long-lived secrets of year three.
• Federate, do not store. Make short-lived federated tokens the default path to cloud and registry targets, and require an exception for every stored static credential.
• Agents are tiered by trust, not by team. Untrusted triggers and production deployments must never share an agent pool or a credential binding.
• Pipeline definitions are production code. Protect them with review, branch rules and approval gates from day one, as OWASP CICD-SEC-1 and CICD-SEC-4 describe.
• Prove it before you scale. A pilot wave with logging, provenance and the health check metrics in place is the gate for every later wave.

The rollout at a glance

The sequence below assumes a platform team owns the server and application teams onboard in waves. Each phase has an exit gate; do not start the next wave until the gate evidence exists.

Phase 0: discovery

OWASP's guidance for CICD-SEC-7 Insecure System Configuration starts with an inventory of systems and versions, each with a named owner. Do the same for everything the new platform will touch before you design it.

1. Pipelines and triggers: list every existing build and release job, what triggers it (push, merge request, schedule, manual, external webhook) and whether untrusted contributors can trigger it.
2. Credentials: list every secret the current build tooling holds, with its target, scope, age and owner. Record metadata only, never values. This list becomes the retirement backlog.
3. Deployment targets: cloud accounts, clusters, registries, package feeds and on-premises hosts, and whether each target can accept federated tokens today.
4. Build hosts: any long-lived build machines, their operating system, who can log in and what network they sit on.
5. Dependencies: shared pipeline libraries, plugins, extensions and third-party actions or steps in use, which OWASP tracks as CICD-SEC-3 Dependency Chain Abuse and CICD-SEC-8 Ungoverned Usage of 3rd Party Services.

The inventory supports NIST CSF 2.0 ID.AM-02 (software, services and systems inventories) and NIST SP 800-53 Rev. 5 CM-8 System Component Inventory. To find leaked values in existing logs and outputs during discovery, follow How to find secrets in CI logs and build artefacts.

Phase 1: design decisions

Hosting model

A delivered service does not remove the customer's configuration duties: OWASP notes that hosted platforms still have to be configured securely. A self-hosted controller adds patching as a hard requirement. A build server exposed to the internet and left unpatched was the entry point in the exploitation of an internet-facing build server by a state intelligence service in 2023, where credentials were dumped from the server once it was reached.

Trust tiers

Define tiers before any team asks for an agent. A workable minimum is three: untrusted (fork and external contributions, no secrets, no internal network), build (trusted branches, read access to dependencies, push to non-production registries) and release (protected branches only, production deployment identities, approval gate). Each tier gets its own agent pool, its own network segment and its own credential bindings. The Self-hosted CI/CD Runners and Build Agents guide covers executor and host hardening for each pool.

Identity model

• People: single sign-on through the corporate identity provider, MFA enforced at the provider, roles mapped from groups. NSA and CISA guidance on defending CI/CD environments recommends federated identity, phishing-resistant authentication, role-based access control and regular audit of administrative accounts.
• Agents: each agent registers with a credential scoped to one pool and expires or is rotated automatically. Registration tokens are secrets, not configuration.
• Deployment jobs: the job requests a signed identity token from the platform, the target exchanges it for short-lived access, and the trust policy on the target binds it to repository, branch or environment claims. For a worked cloud example, see how to set up workload identity federation from a CI system to a cloud provider and the Workload Identity Federation Guide.
• Integrations: the connection to source control and to the artefact registry uses a dedicated service identity with the narrowest scope, never an administrator's personal token.

Network and logging design

Place the controller behind the corporate access layer, not directly on the internet; allow inbound webhooks through a narrow path; give each agent tier egress only to what it needs. Decide now which events go to the SIEM: administrator logins and role changes, credential creation, update and use, agent registration, pipeline definition changes, approvals and deployments. NSA and CISA call for audit logs that record who committed, reviewed and deployed what, when and where.

Phase 2: build and harden the platform

1. Configuration baseline: record every security-relevant setting as code where the platform allows it, and store it in a protected repository. This is your CM-6 Configuration Settings evidence and your rollback point.
2. Administration: create named administrator accounts for a small approved group, separate from day-to-day developer accounts, and remove default or bootstrap accounts. Keep break-glass access in your privileged access tooling.
3. Plugins and extensions: install only what the design needs, from the official source, and record versions. Each plugin runs with the server's privileges.
4. Credential store: configure the platform's secret store or connect an external secrets manager; set scope to project or environment by default, never global. See the Secrets Management Guide.
5. Agent images: build minimal agent images from a pinned base, scan them, sign them and publish them to your internal registry. Ephemeral agents are created from these images per job and destroyed afterwards, matching the NSA and CISA advice to destroy temporary resources after each run.
6. Debug access: restrict interactive and debug access to agents. OWASP warns that anyone who can debug an execution node may see secrets loaded in memory and use the node's identity.
7. Audit pipeline: forward platform audit events and agent logs to the SIEM and confirm delivery with a test event before the pilot begins.
8. Provenance: enable build provenance generation on the hosted, isolated agents so that pilot artefacts already carry signed provenance. SLSA v1.2 Build Level 2 requires a hosted build platform that signs provenance; Build Level 3 adds isolation between runs and keeps signing secrets out of reach of user-defined build steps. See the SLSA and Build Provenance Guide.

Phase 3: pilot

Pick three to five services from teams that volunteer, covering your main languages and at least one production deployment to cloud and one to an internal target. The pilot proves the paved road, not the platform's feature list.

• Templates first: publish shared pipeline templates for build, test, scan, sign and deploy, held in a protected repository with mandatory review. Teams consume the templates; they do not copy them.
• Flow control: apply branch protection to every branch that feeds a deployment and require an approval before production deployment, as OWASP recommends for CICD-SEC-1. Accounts that can push unreviewed code should not be able to trigger deployments.
• Untrusted triggers: run fork and external contributions only in the untrusted tier, with no secrets and no write tokens, to block poisoned pipeline execution (ATT&CK T1677). See the OWASP CI/CD Top 10 CICD-SEC-4 Poisoned Pipeline Execution Guide.
• Third-party steps: pin external actions or steps to an immutable reference, never a moving tag. The pattern is explained in how to pin reusable pipeline steps to commit hashes, and applies to any platform.
• Artefact acceptance: deployment jobs accept only artefacts produced by the platform's own build identity, with provenance verified, matching OWASP's advice to trust only artefacts created by a pre-approved CI service account.

How to test: run the pilot pipelines in parallel with the old build path for two release cycles, compare outputs, then switch production deployment for one service at a time. Rollback: keep the old path available but with its credentials disabled rather than deleted until the pilot exit gate passes; re-enabling a disabled credential is fast, re-creating trust after deletion is not.

Phase 4 and 5: waves, retirement and operation

1. Wave planning: group services by deployment target and trust tier, not by organisation chart. Move high-risk production services in the second or third wave, once templates are proven.
2. Exception process: every stored static credential, persistent agent or bypass of an approval gate needs a ticket with owner, reason and expiry date. Review exceptions monthly.
3. Credential retirement: after each wave, revoke the legacy credentials the wave replaced and record revocation evidence. A credential that still works after migration is the most common leftover.
4. Decommission old build hosts: wipe or destroy them, remove their agent registrations and close their network rules.
5. Hand over to operations: start the quarterly CI/CD server health check after the first wave, and put platform update cadence on the change calendar. NSA and CISA recommend updating CI/CD tools on a regular schedule.

Where identity fits

Almost every credential in a CI/CD platform is a non-human identity: agent registration tokens, integration tokens to source control and registries, deployment identities in cloud accounts and signing keys. OWASP lists Inadequate Identity and Access Management (CICD-SEC-2), Insufficient PBAC (CICD-SEC-5) and Insufficient Credential Hygiene (CICD-SEC-6) separately, and the rollout should track each. Give every pipeline identity an owner and a link to the service it deploys, so it can be reviewed and removed with that service. The source control side of the same trust chain is covered in Securing the source control platform itself.

Small and large organisations

Small teams (one platform engineer, a few dozen repositories) gain most from a delivered service with provider-hosted ephemeral agents and federation to one cloud; phases 0 to 3 can run in six to eight weeks and the trust tiers can be two pools rather than three. Large organisations need a platform team, a template repository with code owners, separate release tiers per regulated business unit, and wave governance with a change board. In both, the approval for production deployment should come from someone other than the author of the change, the separation of duties expected by NIST SP 800-53 AC-5.

Detection and measurement

Alert on: new administrator grants, credentials created at global scope, agent registrations outside approved pools, pipeline definition changes merged without review, deployments without provenance verification, and failed or unusual token exchanges at cloud targets. Track rollout progress with these metrics, then hand them to the health check:

These support NIST CSF 2.0 PR.PS-04 (log records available for continuous monitoring) and PR.PS-06 (secure software development practices integrated and monitored).

Common failure patterns

• Migrating secrets as-is. Copying every stored credential into the new platform on day one recreates the old risk with a new audit trail. Retire and federate instead.
• One shared agent pool. Fork builds and production deployments on the same agents give untrusted code a route to release credentials.
• Templates without owners. Shared templates that anyone can edit become a single point of compromise for every pipeline that imports them.
• Global-scope credentials for convenience. A deployment credential visible to every project turns any project compromise into a production compromise.
• Logging added after go-live. Without audit events from the pilot onwards there is no baseline for what normal administration and deployment look like.
• Old build hosts left running. Decommissioned in name only, they keep their credentials and network access.

Framework mapping

NIST SP 800-204D gives strategies for integrating software supply chain security into CI/CD pipelines, and NIST SP 800-218 (SSDF v1.1) gives the secure development practices the pipeline should enforce.

Practitioner checklist

• Inventory pipelines, triggers, credentials (metadata only), targets and build hosts, each with an owner.
• Record hosting, trust tier, identity, network and logging decisions as approved architecture decisions.
• Connect administrator access to the identity provider with MFA and a small, named admin group.
• Default to project or environment scoped secrets and federated deployment tokens; require exceptions for anything else.
• Build signed, minimal agent images and run jobs on ephemeral agents, separated into trust tiers.
• Publish protected pipeline templates with branch protection and production approval gates.
• Generate and verify build provenance from the pilot onwards.
• Run old and new paths in parallel, disable rather than delete legacy credentials until the gate passes, then revoke.
• Start the quarterly health check after the first wave.

Standards and references

• OWASP: Top 10 CI/CD Security Risks
• OWASP: CICD-SEC-1 Insufficient Flow Control Mechanisms
• OWASP: CICD-SEC-7 Insecure System Configuration
• NSA and CISA: Defending Continuous Integration/Continuous Delivery (CI/CD) Environments
• NIST: SP 800-204D, Strategies for the Integration of Software Supply Chain Security in DevSecOps CI/CD Pipelines
• NIST: SP 800-218, Secure Software Development Framework (SSDF) Version 1.1
• OpenSSF SLSA: Build track basics (v1.2)
• NIST: SP 800-53 Rev. 5, Security and Privacy Controls

Related NHI Mgmt Group resources: CI/CD Security Guide · CI/CD server health check · CI/CD Pipeline Compromise Playbook · CI/CD Pipeline Identity Security Guide · Artefact and container registry health check