A root CA or HSM master key generated without a script, witnesses and a record cannot later be proven trustworthy, and every certificate or key it protects inherits that doubt. A key ceremony is the scripted, multi-person event at which high-value keys are generated, backed up, activated, used for rare operations such as signing a subordinate CA, or destroyed. This template gives [Organisation] a ceremony procedure with named roles, entry criteria, a deviation and abort process, evidence to keep and a review step, laid out in the same numbered structure as the other NHI Mgmt Group playbooks. It is written for PKI engineers, HSM custodians and the auditors who witness them, for both public trust CAs and private enterprise CAs. For the underlying concepts, see the PKI Basics Guide.
How to use this template
- Treat it as a master procedure. Each ceremony then gets its own Key Generation Script (a step-by-step script for that event) derived from sections 3 to 6, approved before the day and signed by every participant on the day.
- Know which rules bind you. A CA in the public web PKI must meet the CA/Browser Forum Baseline Requirements, which require a Key Generation Script for root and externally operated subordinate CA keys and a Qualified Auditor witness or full video recording. A private enterprise CA is bound only by its own Certificate Policy (CP) and Certification Practice Statement (CPS), but following the same discipline makes later audits and incident investigations far easier.
- Replace every placeholder: [Organisation], [HSM model], [M of N], [Facility], [Name, phone] and the CP/CPS section references.
- Keep the incident sections. Sections 4 to 8 cover what happens when a ceremony goes wrong: a seal is broken, a step fails, a card is lost, or a key is suspected compromised. Those are the moments when an unrehearsed team makes irreversible mistakes.
- Rehearse. Run every new script end to end on test hardware with the real participants before the live ceremony.
1. Scope and triggers
1.1 This procedure applies to every operation on [Organisation]'s high-value keys: offline root CA keys, issuing CA keys where the CPS requires it, HSM security world or domain master keys, code signing root keys, DNSSEC key signing keys and any key the key inventory classes as Tier 0.
1.2 A ceremony is required for the following events:
| Event | Typical outputs | Minimum participants |
|---|---|---|
| Root or offline CA key generation | Key pair in HSM, self-signed root certificate, encrypted backups, ceremony record | Ceremony administrator, [M] key custodians, internal witness, external witness or auditor (public CA: Qualified Auditor or full video) |
| Subordinate CA signing | Signed subordinate certificate from a reviewed CSR | Ceremony administrator, [M] key custodians, internal witness |
| CRL or OCSP signer refresh from an offline root | New CRL with next update date, delegated responder certificate | Ceremony administrator, [M] key custodians, internal witness |
| HSM initialisation, addition or replacement | New HSM joined to the security domain, old device zeroised | Ceremony administrator, [M] custodians, internal witness |
| Custodian change or card reissue | New share or card issued, old one destroyed or revoked | Ceremony administrator, [M] custodians, internal witness |
| Key backup restore or disaster recovery | Key restored to replacement HSM in the recovery site | Ceremony administrator, [M] custodians, internal witness |
| Key destruction at end of life | Key and backups destroyed, destruction certificate | Ceremony administrator, [M] custodians, internal witness |
1.3 The guiding standard is NIST SP 800-57 Part 1 Rev. 5, which sets out key states, cryptoperiods and protection requirements across the key lifecycle. Record in the CPS which events [Organisation] treats as ceremonies; DNSSEC root operators, for comparison, hold scheduled ceremonies usually four times a year, chiefly to sign the next quarter's zone signing keys.
2. Roles and contacts
2.1 Every role is a Trusted Role named in the CPS. No person holds two roles in the same ceremony, and the ceremony administrator never holds a key share.
| Role | Named lead and deputy | Responsibilities |
|---|---|---|
| Ceremony administrator | [Name, phone] | Owns the script, reads each step aloud, operates the offline workstation, declares deviations and aborts |
| Key custodians (crypto officers) | [Names, M of N] | Hold activation cards or key shares and PINs; present them only when the script calls for them |
| Internal witness | [Name] | Ticks every step on the master copy, records times, seal numbers and deviations; signs the record |
| External witness or auditor | [Firm, name] | Independently observes and reports (mandatory for public trust root keys under the Baseline Requirements, unless the whole process is video recorded) |
| Safe and facility controller | [Name] | Opens safes and cages, controls room access, logs entry and exit |
| Security operations contact | [Name, phone] | Receives deviation reports and starts the incident process in section 4 if needed |
| PKI policy authority | [Name] | Approves the script before the day and approves any change to it |
2.2 Set the quorum as M of N, for example 3 of 5, so that the ceremony can proceed with one custodian ill or travelling but no single person or pair can act. The Baseline Requirements require multiple person control and split knowledge for CA key generation, and at least dual control for backup, storage and recovery of CA private keys.
3. Detection and triage
For a ceremony, detection means confirming before and during the event that nothing has been tampered with, and triage means deciding whether a discrepancy stops the ceremony.
- Pre-ceremony approval: the policy authority signs the script, the CSR or certificate profile to be signed, and the participant list at least [5] working days before.
- Identity checks: the facility controller checks photo ID for every participant against the approved list and records arrival times.
- Room checks: the room is cleared of phones, laptops and unapproved media. Cameras recording the ceremony are positioned so they never capture PIN entry.
- Seal verification: the witness reads aloud every tamper-evident bag and safe seal number and compares it with the number recorded at the previous ceremony. Any mismatch is a deviation.
- Equipment checks: confirm the HSM serial number, firmware version and its validation (the Baseline Requirements require FIPS 140-2 or 140-3 level 3, or an equivalent Common Criteria evaluation, for CA keys), and the hash of the offline workstation image against the approved value.
- Triage rule: a cosmetic issue (a typo in the script, a late participant) is logged and the ceremony continues; anything that touches integrity (broken seal, wrong serial, hash mismatch, unknown person in the room, missing card) moves to section 4.
4. Containment
4.1 Stop. The ceremony administrator announces a halt, records the time and the step number, and no further commands are entered.
4.2 Secure the state: leave the HSM powered as it is unless the script says otherwise, keep all cards and media on the table in view of the witness, and do not remove anything from the room.
4.3 Decision: the ceremony administrator and the policy authority decide, on the evidence, between three outcomes and record the reason:
- Continue: the cause is explained and harmless, for example a seal number transcribed wrongly last time with photographic proof of the correct one.
- Abort and reschedule: the cause is unexplained but no key material was exposed. Reseal everything in new bags, record the new numbers and close the session.
- Abort and treat as possible compromise: key material, a card with its PIN, or a backup may have been exposed. Call the security operations contact and follow sections 5 and 6.
4.4 For a suspected compromise of an existing CA key, the CA must plan revocation: under the Baseline Requirements an issuing CA revokes a subordinate CA certificate within seven days when it has evidence of key compromise.
5. Eradication
5.1 If a key or share may be exposed, remove it from use: deactivate the key in the HSM, revoke the affected cards or reissue shares to a new quorum, and plan replacement of the key itself if the exposure cannot be ruled out.
5.2 Destroy keys and media by a scripted destruction ceremony: zeroise HSMs using the vendor's documented zeroisation function, physically destroy smartcards and backup tokens, and record serial numbers and the destruction method. The witness signs the destruction certificate.
5.3 For an aborted key generation, zeroise the partly initialised HSM and destroy any backup produced in the session so that no unaccounted copy of an unvalidated key exists.
6. Recovery
- Re-run with a clean start: new script revision, fresh tamper-evident bags, verified hardware, and the same separation of roles.
- Generate and verify: generate the key pair inside the HSM, never on the workstation; record the public key fingerprint; issue the certificate from the approved profile; and verify the certificate fields against the profile line by line before signing anything else.
- Back up: export only encrypted backups or key shares as the HSM design allows, to at least two locations [Facility A, Facility B], under dual control.
- Test the recovery path: at least once a year restore a backup to a spare HSM in a recovery ceremony and confirm a test signature verifies against the published certificate.
- Return to steady state: power down offline equipment, seal workstation, HSM, cards and media in numbered bags, return them to their safes and log the numbers.
7. Communication and notification
- Routine ceremony: publish the new root or subordinate certificate and its fingerprint through the channels in the CPS, and update the key inventory.
- Deviation: report it to the policy authority within [1] working day and include it in the ceremony record given to the auditor.
- Suspected compromise: notify [Organisation]'s incident response lead, relying parties named in the CPS, and, for public trust CAs, the root programmes and any parties required by the CPS, within the timescales they set.
- Customers and internal teams: if certificates must be reissued, give relying teams a schedule and the new chain, because systems that pin the old root will fail.
8. Post-incident review
8.1 After every ceremony, not only failed ones, the ceremony administrator collects the signed script, the witness log, the access logs, the video if recorded, the HSM audit log export and the seal register into one evidence package with a hash, stored for the retention period set in the CPS.
8.2 Within [10] working days, hold a review covering: deviations and their causes, steps that were unclear, timings, and whether any role was missing or doubled.
8.3 For public trust root keys, obtain the Qualified Auditor's report stating whether the CA followed its key ceremony and kept the key pair's integrity and confidentiality, as the Baseline Requirements require.
8.4 Update the master procedure and the next script with what was learned, and record the change in the document history.
Implementation guidance
Choose the level of ceremony to fit the key. The trade-off is cost and scheduling against provable trust:
| Key class | Ceremony level | Notes |
|---|---|---|
| Public trust root or externally operated subordinate CA | Full: script, Qualified Auditor witness or full video, auditor report | Required by the Baseline Requirements section 6.1.1.1 |
| Enterprise offline root CA | Full script, internal and independent witness, video recommended | Lifetime is often 10 to 20 years, so the evidence must outlast staff turnover |
| Enterprise online issuing CA | Script and internal witness for generation and signing | Routine issuance is automated and logged instead |
| Code signing root, DNSSEC KSK, HSM domain master key | Full script and witnesses | Compromise affects every system that trusts the signature |
| Application data keys in a KMS | No ceremony; policy-driven automation | Protect with KMS access policies and rotation instead |
Framework mapping. Record these mappings in the control register:
| Framework | Reference | What the ceremony evidences |
|---|---|---|
| NIST SP 800-53 Rev. 5 | SC-12 Cryptographic Key Establishment and Management, SC-12(1) Availability | Managed key generation and recoverable backups |
| NIST SP 800-53 Rev. 5 | AC-5 Separation of Duties, PE-3 Physical Access Control, AU-10 Non-repudiation, CP-9 System Backup | Distinct roles, controlled room, signed record, tested backups |
| NIST CSF 2.0 | PR.DS-01, PR.AA-05 | Protection of key material at rest; least privilege and separation of duties |
| ISO/IEC 27001:2022 Annex A | A.8.24 Use of cryptography, A.5.3 Segregation of duties | Key management rules and split roles |
| CA/B Forum Baseline Requirements | 5.2.2, 6.1.1.1, 6.2.7 | Dual control, Key Generation Script, validated cryptographic modules |
Small and large organisations. A small team running one private root can meet the spirit with three people (administrator, two custodians on 2 of 3 cards) plus an independent witness from another department and a recorded session. Large CAs keep a standing ceremony team, two geographically separate facilities and a rota of custodians large enough that M of N survives staff loss.
Measure it. Track: ceremonies completed without an integrity deviation; time since the last successful backup restore test; percentage of Tier 0 keys with a current custodian list where every custodian is still employed; seal register mismatches per year (target zero); and days from ceremony to signed evidence package.
Common mistakes to avoid: generating the key on a laptop and importing it to the HSM; letting one administrator also hold a share; storing the PIN with the card in the same bag; never testing a backup restore until a disaster; and custodian lists that still name people who left years ago, which leaves the key unrecoverable. Where machine identities depend on these keys, see the Machine Identity, PKI and Certificate Lifecycle Guide, and for hierarchy design see the Enterprise PKI Design and CA Hierarchy Guide.
Related NHI Mgmt Group resources: PKI Basics Guide · Enterprise PKI Design and CA Hierarchy Guide · Machine Identity, PKI and Certificate Lifecycle Guide · ISO/IEC 27099 PKI Practices and Policy Framework Guide