A SOAR platform decays quietly: a connector's credential expires and enrichment returns blanks, a playbook nobody owns keeps running against a firewall that was replaced, and an exclusion list stops matching the systems it was meant to protect. This health check is a recurring, measurable review of a security orchestration, automation and response (SOAR) platform already in production: its integrations and engines, its playbook estate, the guardrails around automated actions, and the platform as a privileged system. It turns the questions a SOC manager, auditor or CISO would ask into thirteen metrics with formulas and thresholds, so each quarter can be compared with the last. It suits teams looking for a "SOAR audit checklist", "SOAR platform KPIs" or a "playbook health review".
This page measures; it does not explain how to build or harden the platform. For what SOAR is and which use cases to automate, use the SOAR and Security Automation Guide. For admin roles, connector identities, playbook change control and the platform's own audit trail, use Securing the SOAR platform itself. Go back to those pages for every finding this review raises. The check is vendor-neutral and applies to standalone SOAR products, automation built into a SIEM or XDR, and managed automation services.
What this health check covers
The review asks whether the automation the SOC relies on still works, still does only what was agreed, and is still under control. It covers:
- Integration health: whether connectors, engines and relays can still reach the tools they act on, and whether enrichment returns real data rather than empty or error results treated as success.
- Playbook estate: ownership, review currency, dormant playbooks and whether executed actions match the agreed action catalogue.
- Guardrails: protected-target exclusion lists, approval gates for high-impact actions, and whether each action type has been exercised, including its rollback.
- Throughput: whether triggered playbooks start promptly or sit in a queue while an incident develops.
- The platform as a privileged system: connector privilege drift, human access recertification, patch currency and delivery of the platform's own audit log.
Who runs it: the SOAR platform owner or automation engineering lead collects the data; the SOC manager reviews; the CISO signs off. What it is not: a measure of SOC outcomes such as mean time to respond or automation rate, which the SOAR and Security Automation Guide tracks, or of ATT&CK detection coverage, which the SOC Detection Coverage Health Check measures. None of the metrics below double-count those.
Key takeaways
- Empty enrichment is the silent failure. A lookup that times out and returns nothing still lets a playbook "succeed", so analysts and automated decisions act on missing context.
- Every playbook needs a living owner. Playbooks outlast the people and tools they were written for; unowned or dormant ones are where stale logic and stale credentials hide.
- Guardrails drift with the estate. Exclusion lists for critical servers, executives and break-glass accounts must be reconciled with the asset inventory, not set once at rollout.
- An untested undo is not an undo. Each action type in the catalogue should have been exercised, with its rollback, within the last six months.
- The SOAR holds the keys to the estate. Connector identities accumulate permissions they never use; measure the gap between granted and used rights every quarter.
How to use this template
Copy the metrics table and the findings log into your own tracker, then run this SOAR platform health check at the cadence below. Collect each figure from the data source types described, score it against the green, amber and red thresholds, and record every amber or red result as a finding with an owner and a due date. Keep each completed review with its raw extracts, so trends are visible quarter on quarter and an auditor can re-perform any figure.
Cadence and sign-off
| Review | Frequency | Owner | Approver | Evidence kept |
|---|---|---|---|---|
| Connector and engine health test | Daily (automated), reviewed weekly | Automation engineer | SOAR platform owner | Health test results per connector and engine, tickets for failures |
| Playbook estate and action log review | Monthly | Automation engineering lead | SOC manager | Playbook register, execution history, action log joined to the action catalogue |
| Full metrics review (this page) | Quarterly | SOAR platform owner | CISO | Metrics table, raw extracts, findings log |
| Human role and connector permission recertification | Quarterly | Identity governance team | SOC manager | Role list and connector permission list with reviewer decisions |
| Response action exercise programme | Twice a year per action type, and after any major playbook or tool change | Automation engineering lead | Incident response lead | Exercise records showing action, result and rollback evidence |
Metrics
NHI Mgmt Group recommended thresholds; adjust to your risk appetite. "Action catalogue" means the agreed list of every action a playbook may take, with its impact tier and gate, as described in the SOAR platform rollout playbook. "High-impact action" means containment or change actions (isolate a host, disable an account, change a firewall rule, rotate a shared secret).
| ID | Metric | Formula | Data source type | Green | Amber | Red |
|---|---|---|---|---|---|---|
| SOAR-01 | Integration availability | Active connectors, engines and relays that passed every scheduled health test in the period ÷ active connectors, engines and relays | Connector and engine health test history | At least 98% | 90% to below 98% | Below 90%, or any connector used by a containment playbook failing for over 24 hours |
| SOAR-02 | Empty enrichment rate | Enrichment steps that completed but returned no data, a timeout or an error payload ÷ enrichment steps executed | Playbook step-level execution logs | Below 2% | 2% to 10% | Above 10% for any enrichment source |
| SOAR-03 | Playbook ownership and review currency | Production playbooks with a named current owner and a documented review in the last 180 days ÷ production playbooks | Playbook register, content repository history, HR leaver list | At least 95% | 80% to below 95% | Below 80%, or any high-impact playbook with no owner |
| SOAR-04 | Dormant playbooks | Enabled playbooks with no trigger in the last 90 days and no documented reason to stay enabled ÷ enabled playbooks | Playbook register, execution history | Below 5% | 5% to 15% | Above 15% |
| SOAR-05 | Action catalogue conformance | Actions executed in the period that map to a catalogued action at its agreed tier ÷ actions executed | Platform action log, action catalogue | 100% | 98% to below 100%, all explained | Below 98%, or any uncatalogued high-impact action |
| SOAR-06 | Protected-target coverage | Assets and accounts tagged protected (critical servers, executive and break-glass accounts, safety systems) present in the playbook exclusion lists ÷ assets and accounts tagged protected | Asset inventory or CMDB tags, identity inventory, exclusion list export | 100% | 95% to below 100% | Below 95% |
| SOAR-07 | Approval gate integrity | High-impact action executions with a recorded approval by a person other than the playbook author or trigger owner ÷ high-impact action executions that required approval | Approval records, action log, playbook authorship history | 100% | 98% to below 100%, all explained | Below 98%, or any self-approved containment |
| SOAR-08 | Trigger to start latency | Playbook runs that started within 2 minutes of their trigger ÷ playbook runs triggered | Trigger time and run start time in execution history, queue depth metrics | At least 95% | 85% to below 95% | Below 85%, or any containment run queued over 15 minutes |
| SOAR-09 | Action exercise currency | Catalogued action types exercised end to end, including rollback, in the last 6 months ÷ catalogued action types | Exercise records, test case results | At least 90% | 70% to below 90% | Below 70%, or any high-impact action never exercised |
| SOAR-10 | Connector privilege drift | Connector identities holding permissions not used by any catalogued action in the last 90 days ÷ connector identities | Connector permission export, target system audit logs, action catalogue | Below 10% | 10% to 25% | Above 25%, or any connector with tenant-wide or domain-wide admin rights not in the catalogue |
| SOAR-11 | Human access recertification | Human role assignments on the platform reviewed and confirmed in the last quarter ÷ human role assignments | Role assignment export, access review records | 100% | 90% to below 100% | Below 90%, or any leaver still assigned |
| SOAR-12 | Platform patch currency | Platform servers, engines, relays and installed integration packs on a supported version and within the patch window ÷ all such components | Component version inventory, vendor support matrix, patch records | At least 95% | 85% to below 95% | Below 85%, or any component on an unsupported version |
| SOAR-13 | Platform audit log delivery | Days in the period on which the SOAR's own audit log (logins, role changes, playbook edits, credential changes) reached the SIEM or another store its administrators cannot alter ÷ days in the period | SOAR audit log, SIEM or archive receipt records | 100% | 95% to below 100% | Below 95%, or no copy outside the platform |
Notes on the formulas
- SOAR-02: count per enrichment source (reputation, directory, asset, EDR), so one broken lookup is not hidden by healthy ones. Treat a step that returns a default "unknown" verdict because the call failed as empty.
- SOAR-05 and SOAR-07: exclude test-mode runs in a non-production tenant, but include production runs started manually by analysts.
- SOAR-10: work from the target system's audit logs where the platform's own logs do not show which API permissions were exercised.
Where identity fits
Most of the platform's risk is non-human identity risk: every connector is a service account, API key, OAuth client or certificate with standing rights in another system. SOAR-10 and SOAR-11 measure that directly, and the remediation sits with the identity programme as much as the SOC. The Service Account Security Guide and the Secrets Management Guide cover the fixes.
How to collect the data
Use exports and platform telemetry, never a self-assessment. Most of these figures can be saved as reports or queries on the platform, or in the SIEM if the SOAR's execution logs are forwarded there, and rerun each quarter.
- Fix the denominators first. Export the playbook register (name, owner, status, impact tier, last review), the action catalogue, the connector list with each connector's identity, and the protected-target tags from the asset and identity inventories (SOAR-03 to SOAR-06).
- Pull the health test history for every connector, engine and relay over the whole quarter, not just today. A relay that dropped offline for two days in week six matters (SOAR-01).
- Read step-level execution logs for enrichment steps: status, response size and error fields. Group by enrichment source (SOAR-02). The CISA and ASD SIEM and SOAR practitioner guidance warns that a failed query can look like nothing to report; this metric is how to catch that in the automation layer.
- Join the action log to the catalogue. Every executed action should resolve to one catalogue entry and tier. List the ones that do not (SOAR-05). For high-impact actions, join to approval records and to playbook authorship to test separation of duties, in line with NIST SP 800-53 AC-5 (SOAR-07).
- Reconcile exclusion lists with the current asset and identity inventory tags. Compare as sets: tagged but not excluded is the gap (SOAR-06). The Asset Inventory and CMDB Security Guide covers keeping those tags trustworthy.
- Measure queueing from trigger time and run start time in the execution history, with queue depth or worker utilisation metrics where the platform exposes them (SOAR-08).
- Collect exercise evidence: for each action type, the date of the last end-to-end test in production or a representative test environment, the result and evidence that the rollback worked (SOAR-09). The ASD practitioner guidance recommends repeated exercises against well-known techniques plus ad hoc tests of new ones; record which detections triggered each exercise.
- Export connector permissions from each target system (directory roles, cloud role assignments, API scopes, firewall management roles) and compare with the permissions the catalogued actions need and the audit logs show were used (SOAR-10).
- Export human access with assignment dates and the last review decision; check against the leaver list (SOAR-11). Pull component versions for the server, engines, relays and integration packs and compare with the supplier's supported versions (SOAR-12).
- Confirm the audit copy: check that the platform's audit events for each day of the quarter exist in the SIEM or archive, consistent with NIST SP 800-53 AU-9 (SOAR-13).
Red flags that need action now
- A containment playbook enabled while the connector it depends on has failed health tests for more than a day.
- An enrichment source returning empty results for most calls while playbooks keep closing cases as benign.
- A high-impact action executed that is not in the action catalogue, or executed at a lower gate than agreed.
- A containment action approved by the person who wrote or triggered the playbook.
- A protected server, executive account or break-glass account missing from the exclusion lists.
- A playbook owned by a leaver, or written for a tool the organisation has decommissioned, still enabled.
- A connector identity with tenant-wide or domain-wide administrative rights that no catalogued action needs.
- Containment runs sitting in a queue for more than 15 minutes during a live incident.
- No copy of the SOAR's own audit log outside the platform, or a gap of several days in it.
Findings log
Log each amber or red result and each red flag. Carry open items into the next quarter.
| ID | Finding | Metric | Severity | Owner | Due date | Status |
|---|---|---|---|---|---|---|
| HC-SOAR-01 | [Description, e.g. directory enrichment returned empty for 31% of calls after a certificate change] | SOAR-02 | [High] | [Automation engineer] | [Date] | [Open] |
| HC-SOAR-02 | [Description, e.g. 4 new domain controllers not in the isolation exclusion list] | SOAR-06 | [Critical] | [SOAR platform owner] | [Date] | [Open] |
| HC-SOAR-03 | [Description] | [ID] | [Critical / High / Medium / Low] | [Role] | [Date] | [Open / In progress / Closed / Risk accepted] |
Suggested due dates: critical within 7 days, high within 30 days, medium within 90 days. Each risk acceptance needs a named approver and an expiry date.
Remediation map
| Finding type | Where to fix it |
|---|---|
| Failing connectors, empty enrichment, queueing | SOAR and Security Automation Guide (enrichment and case management) |
| Unowned, unreviewed or dormant playbooks; uncatalogued actions | Securing the SOAR platform itself (playbook change control) |
| Exclusion list gaps, approval gate failures, untested rollback | SOAR platform rollout playbook (action classification and gated execution) |
| Connector privilege drift, stale human access | Securing the SOAR platform itself (connector identities, admin roles), Service Account Security Guide |
| Unsupported components, missing audit copy | Securing the SOAR platform itself (hardening and the platform's own audit trail), Patch Management Guide |
| Suspected misuse of the platform or its connectors | If your SOAR platform is compromised |
| Repeated findings that point to replacing the platform | SOAR platform migration playbook |
Framework mapping
| Area | NIST CSF 2.0 | NIST SP 800-53 Rev. 5 | CIS Controls v8.1 |
|---|---|---|---|
| Integration health and throughput (SOAR-01, 02, 08) | DE.AE-02, RS.MA-01 | IR-4, SI-4 | Control 17 |
| Playbook estate and catalogue (SOAR-03 to 05) | PR.PS-01, ID.IM-04 | CM-3, IR-8 | Control 17 |
| Guardrails and exercises (SOAR-06, 07, 09) | RS.MI-01, PR.AA-05 | AC-5, IR-4 | Control 17 |
| Access, patching and self-audit (SOAR-10 to 13) | PR.AA-05 | AC-2, AC-6, IA-5, AU-9 | Control 5, Control 6, Control 7, Control 8 |
Related guides
- Securing the SOAR platform itself: the hardening that SOAR-10 to SOAR-13 test.
- SOAR platform rollout playbook: where the action catalogue, tiers and gates come from.
- SIEM health check: the matching review for the platform that feeds the SOAR its alerts.
- SOC Maturity Model: place the platform results in the wider SOC picture.
- If your SOAR platform is compromised: when a red flag turns out to be an incident.
References
- ASD's ACSC with CISA and partners: Implementing SIEM and SOAR Platforms, Practitioner Guidance
- CISA: Guidance for SIEM and SOAR Implementation
- NIST: SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management
- OASIS: CACAO Security Playbooks Version 2.0
- NIST: SP 800-53 Rev. 5, Security and Privacy Controls
- NIST: Cybersecurity Framework 2.0
- CIS: Critical Security Controls v8.1
Related NHI Mgmt Group resources: SOAR and Security Automation Guide, Securing the SOAR platform itself, SOAR platform rollout playbook, SIEM health check.