A security orchestration, automation and response (SOAR) platform holds working credentials for your EDR, firewalls, email and identity provider and acts on them when an alert fires, so buying one means choosing an automation engine and one of your most privileged non-human identities at the same time. The market now spans standalone SOAR, automation built into SIEM and XDR products, general no-code workflow tools and AI-assisted triage. Products differ less in what they claim than in how playbooks are built and tested, who maintains the integrations, how safely they take response actions and how the bill grows with volume. This vendor-neutral guide is for SOC leads, detection and automation engineers and security architects writing requirements or running an RFP, and ends with a proof of value you can run on your own alerts. For how to design playbooks and approval gates see the SOAR and Security Automation Guide.
Key takeaways
- Decide SIEM-native or standalone first. If most detections and response tools come from one stack, its built-in automation may cover your use cases with fewer stored credentials; standalone SOAR earns its cost across many vendors.
- Integrations are a maintenance contract. Ask who fixes a connector when a vendor changes its API, how fast, and whether you can build your own; the connector count matters less.
- Response safety is a core feature. Approval steps, timeouts, protected-asset lists and one-click reversal decide whether you can automate containment at all, and AI agents need the same gates.
- Price the workload, not the licence. Per-action and per-execution models charge for loops, retries and triggers, so model a month of real alerts before signing.
- Stored credentials make the platform a privileged NHI hub. Per-integration identities, vaulted or federated credentials and controlled playbook editing belong in the evaluation, not the hardening backlog.
Define requirements
Start with work, not features: alert volumes by type, the five use cases that consume most analyst time (often phishing triage, endpoint malware, compromised accounts, leaked secrets and vulnerability ticketing), current time to triage and contain, and who will build playbooks.
- Tool inventory: SIEM, EDR or XDR, email security, identity provider, firewalls and proxies, cloud accounts, ITSM, threat intelligence and any MDR provider. If an MDR service also automates response, agree whose playbooks run, whose credentials they use and who approves. See the SIEM Guide and MDR Guide.
- Authoring skills: analysts building in a visual, low-code editor need guard rails and templates; automation engineers want code steps, Git, testing and CI. Most teams need both.
- Data security workflows: if DLP, data access or insider risk alerts will run through the platform, include their owners, privacy constraints and evidence handling in scope.
- Control obligations: NIST SP 800-61 Rev. 3 names SOAR for analysing and correlating events (DE.AE-02, DE.AE-03) and expects automatic containment for some incidents (RS.MI-01) with manual action available "instead of or in addition to automated containment measures"; NIST SP 800-53 IR-4(1) Automated Incident Handling Processes; CIS Controls v8.1 Control 17; ISO/IEC 27001:2022 Annex A 5.24 to 5.26. CISA, ASD's ACSC and partners published joint executive and practitioner guidance for organisations "seeking to procure" SIEM and SOAR in May 2025.
| Operating model | Best fit | Watch for |
|---|---|---|
| Automation built into the SIEM | One main SIEM; enrichment, assignment and common containment | Weaker cross-vendor case management; workflow engine billed separately |
| XDR or EDR built-in response | Single-vendor endpoint, email and identity stack | Actions limited to that vendor's products |
| Standalone SOAR | Many vendors, large SOC, dedicated automation engineers, cross-tool case management | Integration upkeep, licence cost, another tier-zero system |
| General no-code workflow platform | Fast API automation, enrichment, ticketing | Audit trail, secret storage, approvals and role separation for response actions |
| MDR-provided automation | Small teams outsourcing 24x7 response | Visibility of which actions run automatically and under whose credentials |
A small organisation usually gains most from the automation already in its SIEM or XDR plus phishing triage. A large SOC justifies standalone SOAR when it orchestrates across many vendors, needs one case record per incident and can staff playbook engineering.
Evaluation criteria
Weight the criteria to your use cases and test each one with your own alerts.
| Criterion | What good looks like | Weak signs |
|---|---|---|
| Playbook authoring | Visual editor with code steps, sub-playbooks, versioning, Git export, test runs on sample data, step-level debugging | Edits go live immediately; no history or diff |
| Integrations breadth | Connectors for every tool in your inventory with the read and write actions your playbooks need | Connector exists but only for enrichment, not response |
| Integration maintenance | Stated owner and fix time for broken connectors, custom connector SDK, API version tracking | Community connectors with no support commitment |
| Case management | Cases with tasks, SLA timers, evidence, timeline and approvals; two-way ITSM sync | Cases kept in a separate ticketing tool with no link back |
| AI-assisted triage and agents | Summaries and verdicts with cited evidence, own scoped identity, approval before any write action, logged tool calls | Agent acts with the platform's full credentials; no record of what it read |
| Response action safety | Approval steps with named approvers and timeouts, protected-asset lists, dry run, recorded prior state and undo | Destructive actions with no approval option or no undo |
| Standards | Import and export of OASIS CACAO playbooks; STIX and TAXII for threat intelligence | Proprietary format only, no export |
| Scalability | Queueing and concurrency limits, API rate-limit handling, high availability, on-premises runners for internal networks | Playbooks fail silently under bursts |
| Credential handling | Vault integration or federated identities, one identity per integration, separate read and write credentials, no secrets in playbook bodies | One shared admin key per tool, visible to playbook editors |
| Platform security and audit | SSO and MFA, RBAC separating authors, approvers and publishers, full audit log to the SIEM | Any analyst can edit and publish a live playbook |
| Reporting | Time to triage and contain, automation rate, analyst time saved, approval wait time, failed runs | Run counts only |
SIEM-native automation, CACAO and pricing
SIEM-native automation is now substantial. As one documented example, Microsoft describes Microsoft Sentinel as "also a platform for security orchestration, automation, and response (SOAR)", with automation rules that "Automatically tag, assign, or close incidents without a playbook" and playbooks built on Azure Logic Apps workflows. OASIS CACAO Security Playbooks v2.0 (Committee Specification 01, 27 November 2023) gives a vendor-neutral JSON format with an authentication information object "to abstract that away from the Agents and Targets", and optional digital signatures; ask whether a product can import and export it so playbooks are not locked in.
Common pricing units are analyst seats, actions or executions, playbook runs, connector tiers and platform capacity. Per-execution billing needs modelling: the Logic Apps Consumption plan meters "per execution, not per call", bills triggers even when skipped, counts each action in a loop on every cycle, and bills an action with 5 retries "as 6 executions". Ask for a quote based on a month of your real alert volume, including enrichment loops and retries.
AI triage and agents need approval gates
Alert text, emails and attachments are attacker-controlled input. OWASP warns that "Indirect prompt injections occur when an LLM accepts input from external sources, such as websites or files", and lists excessive functionality, permissions and autonomy as the causes of excessive agency. Its mitigations are what to test: "require a human to approve high-impact actions before they are taken" and "Implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not". See the AI SOC Guide and Prompt Injection Guide.
Where identity fits: the platform's stored credentials
Every integration is a non-human identity, often with write access to isolate hosts, disable users or change firewall rules. Stolen security tooling credentials have real impact: in the BeyondTrust breach a stolen API key led to a major incident at the US Treasury, and ATT&CK tracks T1528 Steal Application Access Token. Federated or managed identities remove stored secrets only where the target supports them; Logic Apps documents managed identities "so you don't have to store credentials, secrets, or access tokens", but only for connectors that "support OAuth with Microsoft Entra". Many third-party integrations still rely on API keys, so require:
- One identity per integration: separate read (enrichment) and write (response) credentials, least-privilege scopes, a named owner and an exportable inventory.
- Vaulted, rotated secrets: retrieved at run time from your vault, never stored in playbook bodies or visible to authors. See the Secrets Management Guide.
- Restricted use: credentials usable only from the platform's egress addresses, with alerts on use from anywhere else.
- Playbook editing as privileged access: anyone who can publish a playbook can run any action its credentials allow, so require review and approval to publish. See the Service Account Security Guide.
Questions to ask vendors
- For each tool in our inventory, which response actions does your connector support, who maintains it, and how quickly was the last breaking API change fixed?
- Show a playbook moving from draft to test to production with review, version history, diff and rollback.
- Which actions can require approval, who can approve, what happens on timeout, and can we protect named assets from automatic action?
- Can you import and export CACAO v2.0 playbooks? If not, how do we export ours if we leave?
- Where are integration credentials stored, can they come from our vault or a federated identity, and can a playbook author read them?
- What identity does your AI assistant or agent use, what can it read and change, and how are its tool calls logged?
- Price our real volume for 12 months: seats, executions, retries, loops, connectors and storage. What happens when we exceed a tier?
- How do you reach systems on internal networks, and what privileges does that runner need?
- Do you support SSO and phishing-resistant MFA at no extra cost, publish an SBOM and run a vulnerability disclosure programme, as the CISA Secure by Demand guide suggests asking?
- If we use an MDR provider, can its analysts run our playbooks under their own audited identities?
Red flags
- Connector counts quoted without action lists or a maintenance commitment.
- Destructive actions with no approval option, no protected-asset list and no undo.
- A shared administrator credential per integration, visible to playbook authors.
- AI agents that take write actions without human approval or a separate identity.
- Pricing that cannot be modelled from your alert volumes, or execution caps that silently stop playbooks.
- No export of playbooks or audit logs.
Proof of concept
- Baseline: record a month of alert volume by type, time to triage, time to contain and analyst hours for three use cases. Agree pass criteria.
- Connect safely: create a dedicated least-privilege identity per integration, store secrets in your vault and confirm authors cannot read them.
- Build both ways: have an analyst build phishing triage in the low-code editor and an engineer build a containment playbook in code with tests in Git.
- Run in shadow mode: replay two weeks of real alerts with actions set to recommend only, and compare with what analysts actually did.
- Test the gates: trigger isolation of a protected server and disablement of a service account; confirm approval, timeout, denial and undo.
- Attack the AI: send a phishing sample containing injected instructions and confirm the assistant cannot take or approve an action because of it.
- Stress and price it: burst ten times normal volume, check queueing and rate limits, and compare the metered usage with the quote.
- Measure and decide: median time to triage and contain, percentage of alerts closed or enriched automatically, approval wait time, failed runs per week and the number of credentials holding write access. Keep these metrics after go-live. See the SOC Guide.
How NHI Mgmt Group can help
We provide independent requirements, RFP and evaluation support. Browse vendors in our products directory or contact us.
Related NHI Mgmt Group resources: SOAR and Security Automation Guide · SIEM Guide · SOC Guide · AI SOC Guide · Secrets Management Guide