Security Automation Maturity Model: Five Levels

Most security teams now run some automation, from enrichment scripts to SOAR playbooks and AI agents, but few can say which actions run without a human, who approved that, how each playbook was tested or how to stop it when it misfires. This model uses NHI Mgmt Group's five levels (Ad hoc, Developing, Defined, Managed and Optimised) to score security automation across six capability areas: scope and action tiers, playbook engineering and testing, governance and approval, automation identities and privilege, safety controls, and measurement. It scores how safely and reliably automation is run, not how much of it exists, which is what separates it from the SOC Maturity Model and the Detection Engineering Maturity Model. It is for SOC managers, automation engineers, incident response leads, CISOs and auditors. For how SOAR and playbooks work, read the SOAR and Security Automation Guide.

Key takeaways

Autonomy is earned per action, not per platform: classify every automated action by impact and reversibility, and let only proven, low-impact, reversible actions run without approval.
Playbooks are code: from Level 3, playbooks live in version control with peer review, test cases and a rollback path, like any production change.
The automation account is a privileged identity: a SOAR platform or AI agent that can disable users and isolate hosts holds some of the most powerful credentials in the estate, and needs scoped, owned, monitored access.
Safety controls come before scale: rate limits, blast-radius caps, allow lists for protected assets and a tested way to pause automation mark the Level 3 line.
AI agents raise the bar, not lower it: joint guidance from six national cyber agencies in May 2026 advises starting agents on low-risk tasks, widening autonomy in phases and requiring human approval for high-impact actions.

The five levels

Level	Name	What it looks like
1	Ad hoc	Individual analysts write scripts and scheduled jobs. No inventory of what runs. Scripts use personal or shared credentials. Nobody reviews changes. If a script fails or misfires, it is noticed by its effects.
2	Developing	A SOAR platform or workflow tool is in use for enrichment, ticketing and notification. Some containment actions automated with analyst click-to-run. Playbooks built in the platform's editor with no version control or testing. A service account with broad rights runs everything.
3	Defined	Written automation standard. Every action classified into tiers by impact and reversibility, with the approval each tier needs. Playbooks in version control, peer reviewed and tested in a non-production environment. Each integration has its own scoped credential and owner. Protected asset lists, rate limits and a pause control in place and tested.
4	Managed	Automation performance measured: actions run, false actions, rollbacks, time saved, approval latency. Playbooks have automated tests run on every change and on a schedule. Credentials short-lived where the target supports it. Automation activity is itself monitored for anomalies. Autonomy for an action is widened only after a measured record of correct decisions.
5	Optimised	Automation is governed as a portfolio: each playbook has an owner, review date, metrics and retirement criteria. AI agents operate within defined autonomy tiers with runtime policy checks and independent monitoring. Exercises and red teaming test whether automation can be tricked or bypass approvals. Lessons from incidents and misfires feed back into playbooks within days.

Capability areas

Score each area separately. A level is reached only when every statement for that level is true for all production automation, not a showcase playbook.

Area	Level 1 Ad hoc	Level 3 Defined	Level 5 Optimised
Scope and action tiers	Whatever someone automated; no inventory	Inventory of all automations; every action assigned to a tier by impact, likelihood of error and reversibility	Tiers reviewed quarterly; actions move between tiers on evidence; retired playbooks removed
Playbook engineering and testing	Scripts on laptops or a shared server	Version control, peer review, test cases for success and failure paths, staging environment, documented rollback	Automated tests in a pipeline; scheduled replay of real alerts; chaos testing of integrations
Governance and approval	No owner, no change control	Named owner per playbook; changes follow change control; approval rules per tier enforced by the platform	Portfolio review with metrics; approval rules set by operators, not by the automation or agent itself
Automation identities and privilege	Personal or shared admin credentials	One identity per integration, scoped to the actions it performs, with an owner and a rotation schedule	Short-lived or just-in-time credentials for high-impact actions; privileged use logged and reviewed; trust reduced on anomalies
Safety controls	None	Protected asset allow lists (domain controllers, executives, critical servers); rate limits and blast-radius caps; tested pause control	Automatic restriction when behaviour deviates from baseline; independent monitors cross-check actions; fail-safe defaults
Measurement	Anecdotes	Counts of runs, failures and manual overrides per playbook	Precision of automated decisions, rollback rate, time saved and approval latency trended and used to set autonomy

Evidence for each level

Level	Acceptable evidence	Not sufficient on its own
2 Developing	Platform playbook list; run history; integration list	A SOAR licence or vendor content pack
3 Defined	Automation standard with action tiers; repository with review history; test records; credential inventory with owners; record of a pause test	Playbooks edited live in production, however well documented
4 Managed	Monthly metrics pack; automated test results; evidence of autonomy decisions based on measured precision; alerts on automation account behaviour	Time-saved figures without error or rollback rates
5 Optimised	Portfolio reviews with retirement decisions; red team reports on automation; examples of misfires turned into playbook changes	Claims of "autonomous SOC" without tier definitions or approval records

Action tiers: deciding what runs without a human

The core of safe automation is a written rule for which actions need which approval. The table is our recommended starting point; adjust it to your risk appetite and record the decision.

Tier (NHI Mgmt Group recommendation)	Typical actions	Reversibility	Approval
Tier 0: read only	Enrichment, reputation lookups, gathering logs, opening tickets	Nothing to reverse	Fully automated
Tier 1: low impact, reversible	Quarantining a single email, blocking a known-bad hash or domain, adding a watchlist entry	Easy, quick	Automated with notification once precision is proven
Tier 2: user or host impact	Isolating an endpoint, revoking sessions, forcing password reset, disabling a standard user	Reversible but disruptive	Analyst approval, or automated for defined high-confidence cases with a cap per hour
Tier 3: high impact or hard to reverse	Disabling privileged or service accounts, firewall changes, deleting data, actions on production infrastructure, actions on logs	Difficult or impossible	Human approval always; two-person rule for the highest-impact actions

The May 2026 joint guidance on agentic AI services from ASD's ACSC, CISA, NSA, the Canadian Centre for Cyber Security, NCSC-NZ and NCSC-UK takes the same approach for AI agents: classify actions by impact, likelihood and reversibility; let system designers or operators, not the agent, decide when approval is needed; add review checkpoints for high-cost actions such as system resets, network egress and deleting critical records; and hold any request to delete logs or audit records until a human approves it.

Playbook engineering and testing

Version control: store playbooks as code or exported definitions in a repository, with peer review before merge. OASIS CACAO Security Playbooks v2.0 (Committee Specification 01, 27 November 2023) gives a vendor-neutral JSON format with required created and modified timestamps, optional revoked and valid_until properties and digital signatures, which makes versioning, expiry and integrity checks explicit. See the OASIS CACAO Security Playbooks Standard Guide.
Test cases: each playbook has tests for the success path, each failure path (integration down, unexpected data, partial completion) and a "should not act" case, such as an alert on a protected asset.
Staging: run new or changed playbooks against a non-production tenant or in a log-only mode that records what it would have done, then compare with analyst decisions before enabling actions.
Rollback: every action that changes state has a documented reverse action, ideally automated, and the playbook records enough to undo it.
Expiry: playbooks carry a review date and are disabled if not reviewed, so integrations and assumptions do not rot silently.

Automation identities and privilege

A SOAR platform or AI agent connected to the identity provider, EDR, firewalls and cloud consoles can disable accounts, isolate hosts and change network rules across the estate. That makes its credentials a high-value target and its mistakes high impact. At Level 3, every integration has its own identity scoped to the actions the playbook performs, with a named owner and a rotation schedule. At Level 4 and above, use short-lived or just-in-time credentials for high-impact actions where the target supports them, log every privileged call, and alert on automation accounts acting outside their usual pattern. The joint agentic AI guidance goes further for agents: treat each agent as a distinct principal with its own keys or certificates, keep a registry and deny unlisted agents, re-verify before privileged calls, prevent agents from changing their own privileges, and reduce an agent's trust when anomalies appear. See the Agentic AI in the SOC Guide and the Detection Engineering for Non-Human Identities Guide.

Safety controls and safe autonomy

Protected asset lists: domain controllers, identity infrastructure, executive accounts, critical servers and break-glass accounts are excluded from automated Tier 2 and Tier 3 actions.
Rate limits and blast-radius caps: limit how many hosts can be isolated or accounts disabled per hour by automation; exceeding the cap pauses the playbook and pages a person.
Pause control: a tested, documented way to stop all automated actions at once, available to the on-call lead, and exercised at least twice a year (our recommendation).
Fail-safe defaults: when data is missing or a decision is uncertain, the automation escalates rather than acts. The joint agentic guidance describes agents configured to stop and escalate to human reviewers in uncertain situations.
Untrusted input: alert content, email bodies and log fields are attacker-controlled. Automation, and especially AI agents, must not treat them as instructions. See OWASP LLM06:2025 Excessive Agency and the AI SOC Guide.
Progressive autonomy: widen an action's tier only after a measured record of correct decisions, and pull autonomy back when failures occur, as the joint guidance recommends for agents.

Measuring automation

Metric	How to calculate	What it shows
Automated action precision	Automated state-changing actions later confirmed correct, divided by all such actions, per playbook	Whether autonomy is justified; the main input to tier decisions
Rollback and override rate	Automated actions reversed or overridden by an analyst, divided by actions taken	Misfires and trust; rising rates mean pull autonomy back
Approval latency	Median minutes from approval request to decision for Tier 2 and 3 actions	Whether approval gates are slowing response enough to need redesign
Coverage of tested playbooks	Production playbooks with passing tests in the last 30 days, divided by all production playbooks	Engineering discipline
Automation identity hygiene	Integration credentials with owner, scope review and rotation in date, divided by all	Privilege risk of the automation estate
Analyst time returned	Estimated minutes saved per run multiplied by runs, net of time spent fixing misfires	Value, stated honestly

Self-assessment questions

Do you have a complete list of every automation, script and agent that can take action in your environment?
Is every automated action assigned to a tier, with the approval that tier needs written down and enforced?
Are playbooks in version control with peer review and tests, or edited live?
Does each integration have its own scoped credential and a named owner?
Which assets are protected from automated action, and when was that list last reviewed?
Can the on-call lead stop all automated actions in under five minutes, and has that been tested?
Do you measure precision and rollback rate per playbook, and use them to set autonomy?
Is the behaviour of automation accounts and AI agents itself monitored?
Could content in an alert or email change what a playbook or agent does?
When a playbook misfires, how long does it take for the fix to reach production?

Score each 0 (no), 1 (partly) or 2 (yes, with evidence). Under 8 suggests Level 1 or 2, 8 to 14 Level 3, above 14 Level 4 or 5. This scoring is our heuristic; the capability table is the authoritative test.

Moving up: roadmap

Level 1 to Level 2

Inventory every script, scheduled job and integration that takes action, with the credentials each uses.
Move recurring scripts onto one orchestration platform; start with read-only enrichment and ticketing.
Replace personal credentials with service identities owned by the team.

Level 2 to Level 3

Write the automation standard: action tiers, approval per tier, protected assets, testing and change rules.
Put playbooks under version control; require peer review and test cases before production.
Split the broad service account into one scoped identity per integration.
Add rate limits, blast-radius caps and a pause control; test the pause in an exercise and record the result.
Run every new state-changing playbook in log-only mode for two to four weeks and compare with analyst decisions before enabling actions; keep the previous version ready to redeploy as the rollback.

Level 3 to Level 4

Publish monthly metrics: precision, rollback rate, approval latency, test coverage and identity hygiene.
Run playbook tests automatically in a pipeline and on a schedule.
Move high-impact actions to short-lived or just-in-time credentials and alert on unusual automation activity.
Widen autonomy only for actions with a measured precision record; document each decision.

Level 4 to Level 5

Govern playbooks as a portfolio with owners, review dates and retirement criteria.
Introduce AI agents within the tier model, with runtime policy checks and independent monitoring.
Red team the automation: test whether crafted alerts, poisoned data or approval bypasses can make it act wrongly.
Feed incident and misfire lessons into playbooks through the normal pipeline within days.

For platform selection, see the SOAR Buyer's Guide and the AI SOC Platform Buyer's Guide.

Framework mapping

Framework	Reference	Relevant levels
NIST SP 800-53 Rev 5	IR-4(1) Automated Incident Handling Processes; IR-4 Incident Handling	Level 2 onwards
NIST SP 800-53 Rev 5	CM-3 Configuration Change Control; CM-4 Impact Analyses (playbooks as changes)	Level 3
NIST SP 800-53 Rev 5	AC-6 Least Privilege; AC-6(9) Log Use of Privileged Functions (automation identities)	Levels 3 and 4
NIST SP 800-53 Rev 5	IR-3 Incident Response Testing (pause and playbook exercises)	Levels 3 to 5
NIST CSF 2.0	RS.MA-03 categorise and prioritise; RS.MI-01 contain; DE.AE-02 analyse adverse events	Actions automated in tiers
NIST CSF 2.0	PR.AA-05 least privilege and separation of duties; ID.IM-02 improvements from tests and exercises	Levels 3 to 5
NIST SP 800-61 Rev 3	Incident response as a CSF 2.0 Community Profile (final, April 2025)	All levels
OWASP Top 10 for LLM Applications 2025	LLM06 Excessive Agency	AI agents at Levels 4 and 5

Common failure patterns

One god account: the orchestration platform runs every integration under one service account with domain or tenant admin rights, so any playbook bug or platform compromise reaches everything.
Automating before tiering: containment actions switched on to hit a time-to-respond target, with no protected asset list, until automation isolates a critical server or disables a service account that production depends on.
Live editing: playbooks changed in the production editor with no review or test, so a small change silently breaks a branch that only runs during an incident.
No off switch: nobody knows how to stop a misfiring playbook quickly, so it runs until the rate of damage is noticed.
Time saved, errors ignored: dashboards report hours saved but not misfires and rollbacks, so autonomy expands on a one-sided measure.
Agents trusting their inputs: AI triage agents act on instructions hidden in alert or email content.

Lessons from incidents

The PocketOS Database Deletion 2026 involved an AI coding agent rather than a SOC playbook, but the lessons transfer directly to security automation. Working on a staging task, the agent found an API token meant only for managing custom domains that in fact had account-wide authority, and used it to delete a production database and its volume-level backups in one call. The customer-side controls that would have contained it are the ones this model scores: one scoped credential per purpose and environment, human approval for destructive (Tier 3) actions, and backups kept outside the reach of the credentials automation holds.

Practitioner checklist

Inventory every automation, script and agent that can take action, with its credentials and owner.
Write action tiers by impact and reversibility, and enforce the approval each tier needs in the platform.
Keep playbooks in version control with peer review, test cases and a documented rollback.
Give each integration its own scoped identity with an owner and rotation schedule.
Maintain protected asset lists and exclude them from automated high-impact actions.
Set rate limits and blast-radius caps, and test a pause control at least twice a year.
Run new state-changing playbooks in log-only mode before enabling actions.
Measure precision and rollback rate per playbook and use them to widen or narrow autonomy.
Monitor automation accounts and AI agents as privileged identities.
Require human approval for any automated or agent request to delete logs or audit records.

Security Automation Maturity Model: From Scripts to Governed Automated Response