Join our Newsletter — 33% off our NHI Course
Home› Guides› SOAR platform health check: 13 metrics with thresholds…
Template SOC & Detection Engineering

SOAR platform health check: 13 metrics with thresholds to audit quarterly

PDF for newsletter subscribers · free to read online

← All guides
By NHI Mgmt Group, edited by Lalit Choda Updated 14 November 2026 13 min read
On this page

A SOAR platform decays quietly: a connector's credential expires and enrichment returns blanks, a playbook nobody owns keeps running against a firewall that was replaced, and an exclusion list stops matching the systems it was meant to protect. This health check is a recurring, measurable review of a security orchestration, automation and response (SOAR) platform already in production: its integrations and engines, its playbook estate, the guardrails around automated actions, and the platform as a privileged system. It turns the questions a SOC manager, auditor or CISO would ask into thirteen metrics with formulas and thresholds, so each quarter can be compared with the last. It suits teams looking for a "SOAR audit checklist", "SOAR platform KPIs" or a "playbook health review".

This page measures; it does not explain how to build or harden the platform. For what SOAR is and which use cases to automate, use the SOAR and Security Automation Guide. For admin roles, connector identities, playbook change control and the platform's own audit trail, use Securing the SOAR platform itself. Go back to those pages for every finding this review raises. The check is vendor-neutral and applies to standalone SOAR products, automation built into a SIEM or XDR, and managed automation services.

What this health check covers

The review asks whether the automation the SOC relies on still works, still does only what was agreed, and is still under control. It covers:

  • Integration health: whether connectors, engines and relays can still reach the tools they act on, and whether enrichment returns real data rather than empty or error results treated as success.
  • Playbook estate: ownership, review currency, dormant playbooks and whether executed actions match the agreed action catalogue.
  • Guardrails: protected-target exclusion lists, approval gates for high-impact actions, and whether each action type has been exercised, including its rollback.
  • Throughput: whether triggered playbooks start promptly or sit in a queue while an incident develops.
  • The platform as a privileged system: connector privilege drift, human access recertification, patch currency and delivery of the platform's own audit log.

Who runs it: the SOAR platform owner or automation engineering lead collects the data; the SOC manager reviews; the CISO signs off. What it is not: a measure of SOC outcomes such as mean time to respond or automation rate, which the SOAR and Security Automation Guide tracks, or of ATT&CK detection coverage, which the SOC Detection Coverage Health Check measures. None of the metrics below double-count those.

Key takeaways

  • Empty enrichment is the silent failure. A lookup that times out and returns nothing still lets a playbook "succeed", so analysts and automated decisions act on missing context.
  • Every playbook needs a living owner. Playbooks outlast the people and tools they were written for; unowned or dormant ones are where stale logic and stale credentials hide.
  • Guardrails drift with the estate. Exclusion lists for critical servers, executives and break-glass accounts must be reconciled with the asset inventory, not set once at rollout.
  • An untested undo is not an undo. Each action type in the catalogue should have been exercised, with its rollback, within the last six months.
  • The SOAR holds the keys to the estate. Connector identities accumulate permissions they never use; measure the gap between granted and used rights every quarter.

How to use this template

Copy the metrics table and the findings log into your own tracker, then run this SOAR platform health check at the cadence below. Collect each figure from the data source types described, score it against the green, amber and red thresholds, and record every amber or red result as a finding with an owner and a due date. Keep each completed review with its raw extracts, so trends are visible quarter on quarter and an auditor can re-perform any figure.

Cadence and sign-off

ReviewFrequencyOwnerApproverEvidence kept
Connector and engine health testDaily (automated), reviewed weeklyAutomation engineerSOAR platform ownerHealth test results per connector and engine, tickets for failures
Playbook estate and action log reviewMonthlyAutomation engineering leadSOC managerPlaybook register, execution history, action log joined to the action catalogue
Full metrics review (this page)QuarterlySOAR platform ownerCISOMetrics table, raw extracts, findings log
Human role and connector permission recertificationQuarterlyIdentity governance teamSOC managerRole list and connector permission list with reviewer decisions
Response action exercise programmeTwice a year per action type, and after any major playbook or tool changeAutomation engineering leadIncident response leadExercise records showing action, result and rollback evidence

Metrics

NHI Mgmt Group recommended thresholds; adjust to your risk appetite. "Action catalogue" means the agreed list of every action a playbook may take, with its impact tier and gate, as described in the SOAR platform rollout playbook. "High-impact action" means containment or change actions (isolate a host, disable an account, change a firewall rule, rotate a shared secret).

IDMetricFormulaData source typeGreenAmberRed
SOAR-01Integration availabilityActive connectors, engines and relays that passed every scheduled health test in the period ÷ active connectors, engines and relaysConnector and engine health test historyAt least 98%90% to below 98%Below 90%, or any connector used by a containment playbook failing for over 24 hours
SOAR-02Empty enrichment rateEnrichment steps that completed but returned no data, a timeout or an error payload ÷ enrichment steps executedPlaybook step-level execution logsBelow 2%2% to 10%Above 10% for any enrichment source
SOAR-03Playbook ownership and review currencyProduction playbooks with a named current owner and a documented review in the last 180 days ÷ production playbooksPlaybook register, content repository history, HR leaver listAt least 95%80% to below 95%Below 80%, or any high-impact playbook with no owner
SOAR-04Dormant playbooksEnabled playbooks with no trigger in the last 90 days and no documented reason to stay enabled ÷ enabled playbooksPlaybook register, execution historyBelow 5%5% to 15%Above 15%
SOAR-05Action catalogue conformanceActions executed in the period that map to a catalogued action at its agreed tier ÷ actions executedPlatform action log, action catalogue100%98% to below 100%, all explainedBelow 98%, or any uncatalogued high-impact action
SOAR-06Protected-target coverageAssets and accounts tagged protected (critical servers, executive and break-glass accounts, safety systems) present in the playbook exclusion lists ÷ assets and accounts tagged protectedAsset inventory or CMDB tags, identity inventory, exclusion list export100%95% to below 100%Below 95%
SOAR-07Approval gate integrityHigh-impact action executions with a recorded approval by a person other than the playbook author or trigger owner ÷ high-impact action executions that required approvalApproval records, action log, playbook authorship history100%98% to below 100%, all explainedBelow 98%, or any self-approved containment
SOAR-08Trigger to start latencyPlaybook runs that started within 2 minutes of their trigger ÷ playbook runs triggeredTrigger time and run start time in execution history, queue depth metricsAt least 95%85% to below 95%Below 85%, or any containment run queued over 15 minutes
SOAR-09Action exercise currencyCatalogued action types exercised end to end, including rollback, in the last 6 months ÷ catalogued action typesExercise records, test case resultsAt least 90%70% to below 90%Below 70%, or any high-impact action never exercised
SOAR-10Connector privilege driftConnector identities holding permissions not used by any catalogued action in the last 90 days ÷ connector identitiesConnector permission export, target system audit logs, action catalogueBelow 10%10% to 25%Above 25%, or any connector with tenant-wide or domain-wide admin rights not in the catalogue
SOAR-11Human access recertificationHuman role assignments on the platform reviewed and confirmed in the last quarter ÷ human role assignmentsRole assignment export, access review records100%90% to below 100%Below 90%, or any leaver still assigned
SOAR-12Platform patch currencyPlatform servers, engines, relays and installed integration packs on a supported version and within the patch window ÷ all such componentsComponent version inventory, vendor support matrix, patch recordsAt least 95%85% to below 95%Below 85%, or any component on an unsupported version
SOAR-13Platform audit log deliveryDays in the period on which the SOAR's own audit log (logins, role changes, playbook edits, credential changes) reached the SIEM or another store its administrators cannot alter ÷ days in the periodSOAR audit log, SIEM or archive receipt records100%95% to below 100%Below 95%, or no copy outside the platform

Notes on the formulas

  • SOAR-02: count per enrichment source (reputation, directory, asset, EDR), so one broken lookup is not hidden by healthy ones. Treat a step that returns a default "unknown" verdict because the call failed as empty.
  • SOAR-05 and SOAR-07: exclude test-mode runs in a non-production tenant, but include production runs started manually by analysts.
  • SOAR-10: work from the target system's audit logs where the platform's own logs do not show which API permissions were exercised.

Where identity fits

Most of the platform's risk is non-human identity risk: every connector is a service account, API key, OAuth client or certificate with standing rights in another system. SOAR-10 and SOAR-11 measure that directly, and the remediation sits with the identity programme as much as the SOC. The Service Account Security Guide and the Secrets Management Guide cover the fixes.

How to collect the data

Use exports and platform telemetry, never a self-assessment. Most of these figures can be saved as reports or queries on the platform, or in the SIEM if the SOAR's execution logs are forwarded there, and rerun each quarter.

  1. Fix the denominators first. Export the playbook register (name, owner, status, impact tier, last review), the action catalogue, the connector list with each connector's identity, and the protected-target tags from the asset and identity inventories (SOAR-03 to SOAR-06).
  2. Pull the health test history for every connector, engine and relay over the whole quarter, not just today. A relay that dropped offline for two days in week six matters (SOAR-01).
  3. Read step-level execution logs for enrichment steps: status, response size and error fields. Group by enrichment source (SOAR-02). The CISA and ASD SIEM and SOAR practitioner guidance warns that a failed query can look like nothing to report; this metric is how to catch that in the automation layer.
  4. Join the action log to the catalogue. Every executed action should resolve to one catalogue entry and tier. List the ones that do not (SOAR-05). For high-impact actions, join to approval records and to playbook authorship to test separation of duties, in line with NIST SP 800-53 AC-5 (SOAR-07).
  5. Reconcile exclusion lists with the current asset and identity inventory tags. Compare as sets: tagged but not excluded is the gap (SOAR-06). The Asset Inventory and CMDB Security Guide covers keeping those tags trustworthy.
  6. Measure queueing from trigger time and run start time in the execution history, with queue depth or worker utilisation metrics where the platform exposes them (SOAR-08).
  7. Collect exercise evidence: for each action type, the date of the last end-to-end test in production or a representative test environment, the result and evidence that the rollback worked (SOAR-09). The ASD practitioner guidance recommends repeated exercises against well-known techniques plus ad hoc tests of new ones; record which detections triggered each exercise.
  8. Export connector permissions from each target system (directory roles, cloud role assignments, API scopes, firewall management roles) and compare with the permissions the catalogued actions need and the audit logs show were used (SOAR-10).
  9. Export human access with assignment dates and the last review decision; check against the leaver list (SOAR-11). Pull component versions for the server, engines, relays and integration packs and compare with the supplier's supported versions (SOAR-12).
  10. Confirm the audit copy: check that the platform's audit events for each day of the quarter exist in the SIEM or archive, consistent with NIST SP 800-53 AU-9 (SOAR-13).

Red flags that need action now

  • A containment playbook enabled while the connector it depends on has failed health tests for more than a day.
  • An enrichment source returning empty results for most calls while playbooks keep closing cases as benign.
  • A high-impact action executed that is not in the action catalogue, or executed at a lower gate than agreed.
  • A containment action approved by the person who wrote or triggered the playbook.
  • A protected server, executive account or break-glass account missing from the exclusion lists.
  • A playbook owned by a leaver, or written for a tool the organisation has decommissioned, still enabled.
  • A connector identity with tenant-wide or domain-wide administrative rights that no catalogued action needs.
  • Containment runs sitting in a queue for more than 15 minutes during a live incident.
  • No copy of the SOAR's own audit log outside the platform, or a gap of several days in it.

Findings log

Log each amber or red result and each red flag. Carry open items into the next quarter.

IDFindingMetricSeverityOwnerDue dateStatus
HC-SOAR-01[Description, e.g. directory enrichment returned empty for 31% of calls after a certificate change]SOAR-02[High][Automation engineer][Date][Open]
HC-SOAR-02[Description, e.g. 4 new domain controllers not in the isolation exclusion list]SOAR-06[Critical][SOAR platform owner][Date][Open]
HC-SOAR-03[Description][ID][Critical / High / Medium / Low][Role][Date][Open / In progress / Closed / Risk accepted]

Suggested due dates: critical within 7 days, high within 30 days, medium within 90 days. Each risk acceptance needs a named approver and an expiry date.

Remediation map

Finding typeWhere to fix it
Failing connectors, empty enrichment, queueingSOAR and Security Automation Guide (enrichment and case management)
Unowned, unreviewed or dormant playbooks; uncatalogued actionsSecuring the SOAR platform itself (playbook change control)
Exclusion list gaps, approval gate failures, untested rollbackSOAR platform rollout playbook (action classification and gated execution)
Connector privilege drift, stale human accessSecuring the SOAR platform itself (connector identities, admin roles), Service Account Security Guide
Unsupported components, missing audit copySecuring the SOAR platform itself (hardening and the platform's own audit trail), Patch Management Guide
Suspected misuse of the platform or its connectorsIf your SOAR platform is compromised
Repeated findings that point to replacing the platformSOAR platform migration playbook

Framework mapping

AreaNIST CSF 2.0NIST SP 800-53 Rev. 5CIS Controls v8.1
Integration health and throughput (SOAR-01, 02, 08)DE.AE-02, RS.MA-01IR-4, SI-4Control 17
Playbook estate and catalogue (SOAR-03 to 05)PR.PS-01, ID.IM-04CM-3, IR-8Control 17
Guardrails and exercises (SOAR-06, 07, 09)RS.MI-01, PR.AA-05AC-5, IR-4Control 17
Access, patching and self-audit (SOAR-10 to 13)PR.AA-05AC-2, AC-6, IA-5, AU-9Control 5, Control 6, Control 7, Control 8

References

Related NHI Mgmt Group resources: SOAR and Security Automation Guide, Securing the SOAR platform itself, SOAR platform rollout playbook, SIEM health check.

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Researched and written by the NHI Mgmt Group editorial team using AI tools, under the editorial direction of Lalit Choda. Facts are drawn from the sources listed on this page and checked by our automated quality process. Last updated 14 November 2026. Spotted an error? Use our contact page.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.