Compliance Automation Guide

Most compliance programmes still prove their controls with screenshots, spreadsheets and evidence requests gathered weeks before an audit, so they find control failures late and spend audit season rebuilding history. Compliance automation replaces that with control tests that run against live systems, evidence collected from source with its query and timestamp, guardrails written as code, and control data held in machine-readable formats such as NIST OSCAL. This guide covers continuous control monitoring, evidence automation, policy as code, OSCAL, GRC integrations, automated access reviews, auditor workflows and the limits of automation. It is for security, GRC and platform engineering teams running SOC 2, ISO/IEC 27001, NIST SP 800-53 or FedRAMP programmes.

Key takeaways

• Automate the test, not just the evidence: NIST IR 8011 frames automated assessment as comparing a desired state with an actual state; write each control test that way, with a pass condition, a frequency and an owner.
• Prevent where you can, detect where you must: policy as code (OPA and Rego, AWS Config rules, Azure Policy) stops non-compliant changes before deployment; continuous monitoring catches drift afterwards.
• Keep control data machine-readable: OSCAL expresses catalogues, baselines, system security plans, assessment results and POA&Ms in JSON, XML or YAML, and FedRAMP is moving packages to machine-readable formats.
• Integrations are privileged non-human identities: a compliance platform holds read access to cloud, identity and code systems; scope, vault and monitor those credentials.
• Automation does not equal compliance: design judgement, policy approval, risk acceptance and the auditor's own testing stay human, and the AICPA has warned firms to evaluate SOC tool arrangements carefully.

What compliance automation covers

"Compliance automation" is used for five different capabilities. Treat them separately, because they fail in different ways.

Selecting a platform is covered in the GRC Platform Buyer's Guide; this guide is about running the capabilities well, whichever tooling you use.

Continuous control monitoring

NIST SP 800-137 (September 2011) describes information security continuous monitoring (ISCM) as a strategy and programme that gives visibility into assets, awareness of threats and vulnerabilities, and visibility into the effectiveness of deployed controls. It sets a six-step cycle: define the strategy, establish the programme, implement it, analyse and report, respond, and review and update. In SP 800-53 Rev. 5 the matching control is CA-7 Continuous Monitoring, with CA-7(4) adding risk monitoring.

NIST IR 8011 Volume 1 (June 2017) supplies the testing model. Automated assessments are "defect checks" using the SP 800-53A test method, "comparing a desired and actual state (or behavior)", and controls are broken into determination statements so each one can be tested. That is the discipline most CCM programmes lack: a test called "MFA enabled" with no defined population, pass condition or exception rule produces numbers nobody can defend.

Anatomy of a defensible control test

• Control and determination statement: the internal control ID plus the framework requirement it evidences (for example SP 800-53 IA-2(1), ISO/IEC 27001 A.8.5, SOC 2 CC6.1).
• Population: the authoritative list being tested (all human accounts in the identity provider, all production accounts in the cloud organisation), and how completeness is proven.
• Desired state: a precise, testable condition, such as "every enabled human account has a phishing-resistant MFA method registered".
• Query and source: the exact API call or query, stored with the result so the auditor can re-perform it.
• Frequency: continuous or daily for configuration, monthly or quarterly for process controls, set by risk rather than tool defaults.
• Exception handling: approved exceptions with an owner and expiry, kept separate from failures.
• Owner and response: who fixes a failure, within what time, and when it becomes a POA&M item (SP 800-53 CA-5).

Examples of automatable tests

The cross-framework column is a starting point; maintain the real relationships in a common control framework, as described in the Control Mapping Guide.

Automating evidence collection

Automated evidence is only better than a screenshot if an auditor can trust where it came from. Each evidence object should carry the source system, the query or API call, the collecting identity, a UTC timestamp and a content hash, and it should be stored write-once for the audit period.

• Prove completeness: SOC 2 Type 2 and ISO 27001 auditors sample from populations. Export the full population (all changes, all new joiners) from source and reconcile counts against a second system, or the sample is worthless.
• Keep history, not just state: a dashboard showing "100% compliant today" does not evidence the audit period. Store every test result so you can show when a control failed and how quickly it was fixed.
• Collect what proves operation: for process controls, collect the artefacts the process generates (approved tickets, signed review campaigns, incident post-mortems) rather than a statement that the process exists.
• Automate policy acknowledgement carefully: tie acknowledgement to the HR roster so new joiners are chased automatically, and record the policy version acknowledged.
• Leave judgement to people: interviews, walkthroughs, management review and risk acceptance still need named humans and dated records.

Policy as code: preventive guardrails

Policy as code turns a control requirement into an executable rule evaluated at the point of change. The Open Policy Agent, a graduated CNCF project, "decouples policy decision-making from policy enforcement": software queries OPA with structured JSON input and receives a decision computed by policies written in the declarative Rego language. Common uses are Kubernetes admission control, Terraform plan validation and CI/CD gates.

package terraform.storage

deny contains msg if {
  some rc in input.resource_changes
  rc.type == "aws_s3_bucket_public_access_block"
  rc.change.after.block_public_acls == false
  msg := sprintf("%s must block public ACLs (CM-6)", [rc.address])
}

• AWS Config: rules "represent your ideal configuration settings"; results are COMPLIANT, NON_COMPLIANT, NOT_APPLICABLE or INSUFFICIENT_DATA. A conformance pack is "a collection of AWS Config rules and remediation actions" deployed as one unit to an account and Region or across an organisation.
• Azure Policy: evaluation order matters. Microsoft documents that append and modify run before deny, and deny before audit so an undesired resource is not logged twice, while auditIfNotExists and deployIfNotExists evaluate after the resource provider returns success.

Cloud posture tooling that runs many of these checks is covered in the CNAPP Guide.

OSCAL: machine-readable controls and system security plans

NIST's Open Security Controls Assessment Language (OSCAL) provides "open, machine-readable formats available in XML, JSON, and YAML" in three layers.

• FedRAMP: RFC-0024 ("FedRAMP Rev5 Machine-Readable Packages", marked as proposed on fedramp.gov) lists OSCAL as an approved format "so long as this project is maintained", would require initial packages in a machine-readable format from 30 September 2026, and proposes revocation after a grace period ending 30 September 2027. It also says providers should use machine-generated telemetry instead of hand-written narratives where feasible.
• FedRAMP 20x: replaces document-heavy assessment with Key Security Indicators that providers measure continuously; FedRAMP says status "should be automatically enforced and validated whenever possible" and that it will stop accepting new Rev5 certifications on 11 June 2027. See the FedRAMP Guide.
• Cloud Security Alliance: the Cloud Controls Matrix v4.1 (197 control objectives in 17 domains, now combined with the CAIQ) publishes implementation guidelines in JSON, YAML and OSCAL.

Even outside FedRAMP, holding the SSP and control set in OSCAL makes framework updates a diff rather than a rewrite. See the NIST SP 800-53 Guide for the catalogue itself.

Where identity fits: integrations and automated access reviews

Compliance automation is an identity problem in two directions.

• The platform's own credentials: to collect evidence, a compliance or GRC platform holds API tokens, OAuth grants or cross-account roles with read access to your cloud organisation, identity provider, HR system and code repositories. That is a high-value non-human identity. Use read-only, scoped roles with external IDs or workload identity federation instead of long-lived keys, store any secrets in a vault, rotate them, and alert on use from unexpected networks. The Salesloft Drift breach showed how OAuth tokens held by one integration opened hundreds of customer tenants. See the Service Account Security Guide.
• Access reviews: automation should build complete review populations from source (including service accounts, API keys and OAuth apps, not just human users), route them to the right owner, revoke on rejection and evidence the revocation. It cannot make the reviewer's decision meaningful; watch approval rates and time-to-decision for rubber-stamping. See the Access Reviews Guide.
• Identity tests pay back first: MFA coverage, leaver deprovisioning, privileged group membership and dormant non-human identities map directly to SOC 2 CC6 and ISO 27001 A.5.15 to A.5.18, and they are fully automatable.

Audit readiness and the auditor's workflow

• Give auditors re-performable evidence: read-only auditor access to evidence with its query, source and timestamp lets the auditor re-perform tests rather than accept screenshots. Agree the evidence format with the audit firm before the period starts.
• The auditor still tests: a platform's green status is management's assertion, not the auditor's conclusion. Expect the auditor to select samples, inspect source systems and test design. See the SOC 2 Guide and the ISO 27001 Certification Guide.
• Keep independence clean: the AICPA says it is looking into anonymously published allegations about a compliance vendor offering SOC services and that "SOC services should be thoroughly evaluated by service organizations and CPA firms". Its ethics staff (April 2026) warn that business arrangements with SOC tool providers "can raise significant threats" to compliance with the Code of Professional Conduct. Choose the auditor separately from the platform, and be wary of bundled "audit included" offers.
• Write the system description yourself: templates speed drafting, but the description and control narratives must match how your system actually works.

Choosing an approach

• Small organisation, first SOC 2 or ISO 27001: a compliance automation platform with prebuilt integrations and tests is usually faster than building. Budget time to tune tests to your environment and to write policies that describe what you really do.
• Cloud-native engineering organisation: keep preventive policy in your own pipeline (OPA, cloud-native rules) and let the platform consume the results, so guardrails survive a tool change.
• Large or regulated organisation: run a common control framework with stable IDs, store control data in OSCAL, feed CCM results from existing tools (CSPM, SIEM, IGA, vulnerability management) and use the GRC platform for workflow and reporting.
• Federal market: plan for machine-readable FedRAMP packages and 20x indicators now; retrofitting OSCAL onto a Word SSP is expensive.

How to implement

1. Fix the control set first: agree the frameworks in scope and a common control list with owners; automation built on an unmapped control set multiplies confusion.
2. Connect sources read-only: identity provider, HR, cloud organisation, code, ticketing and endpoint management, each with a scoped, inventoried integration identity.
3. Turn on tests in monitor-only mode: run for two to four weeks, triage false positives, fix populations and record approved exceptions before any result is reported upwards.
4. Add preventive guardrails in audit mode: deploy Azure Policy with the audit effect before deny, Gatekeeper constraints with a dry-run or warn enforcement action, and AWS Config rules in detective mode before proactive gating. Promote one rule at a time to blocking in a pilot account or subscription.
5. Plan the roll-back: keep every policy in version control so a blocking rule can be reverted to audit by a single change, and give teams a documented exception path.
6. Export and rehearse: export control data and evidence (OSCAL where supported) and run a mock audit with your audit firm on one control family.
7. Review quarterly: retire tests nobody acts on, add tests for new systems, and re-check integration permissions.

Detection and measurement

• Automated coverage: share of in-scope controls with at least one automated test, reported separately for preventive and detective tests.
• Mean time to remediate failures: from first failing test to passing, by control family, against your SLA.
• Test accuracy: false positive rate found during triage, aiming to fall each quarter.
• Exception health: open exceptions past expiry (target zero).
• Audit outcome: auditor-identified exceptions not already flagged by your own monitoring (target zero; each one is a monitoring gap).
• Integration hygiene: compliance integrations with write permissions or credentials older than the rotation policy.

Framework mapping

Common failure patterns

• Green dashboard, wrong population: the MFA test covers the identity provider, but local accounts in SaaS applications and cloud root users sit outside it. Reconcile populations against a second source.
• Policies copied from templates: automatically generated policies describe controls the organisation does not run, and the auditor tests the policy as written.
• Over-privileged integrations: the compliance platform is granted administrator roles "to make setup easier", turning a monitoring tool into a breach path. Keep it read-only and treat its tokens as the Salesloft Drift case warns.
• Blocking too early: switching a deny policy on across production without an audit period breaks deployments and leads teams to demand blanket exemptions.
• State without history: keeping only current status, so the organisation cannot show when a control failed during the period or that it was fixed.
• Treating the platform as the auditor: assuming a platform-bundled audit is equivalent to an independent one, which the AICPA's notices suggest auditees should check carefully.

Practitioner checklist

• Define every automated test with a determination statement, population, desired state, query, frequency, owner and exception rule.
• Store test results and evidence with source, query, collecting identity, timestamp and hash, retained for the audit period.
• Prove population completeness by reconciling against a second system.
• Run new tests and guardrails in monitor or audit mode before reporting or blocking.
• Keep policy as code in version control with a one-change roll-back to audit mode.
• Give every compliance integration a read-only, scoped, inventoried identity and rotate its credentials.
• Include non-human identities in automated access review populations.
• Hold control data and the SSP in OSCAL where your tools support it, especially for FedRAMP.
• Choose and contract the auditor independently of the automation platform.

Standards and references

• NIST: SP 800-137, Information Security Continuous Monitoring (ISCM) for Federal Information Systems and Organizations
• NIST: IR 8011 Volume 1, Automation Support for Security Control Assessments: Overview
• NIST: OSCAL Layers and Models
• FedRAMP: RFC-0024 FedRAMP Rev5 Machine-Readable Packages
• FedRAMP: FedRAMP 20x
• Cloud Security Alliance: Cloud Controls Matrix
• Open Policy Agent: Documentation
• AICPA and CIMA: Ethics Staff Insights, Business Arrangements with SOC Tool Providers

Related NHI Mgmt Group resources: GRC Platform Buyer's Guide · Control Mapping Guide · Access Reviews Guide · SOC 2 Guide