Control Design vs Operating Effectiveness Testing

A control can be perfectly designed on paper and still fail every week in practice, and auditors test those two things separately. Design testing asks whether a control, run as described, would actually meet its objective. Operating effectiveness testing asks whether it really ran that way, consistently, across a period, performed by someone with the authority and competence to do it. This guide explains what each test involves, the evidence an auditor expects, how sampling and deficiency classification work, and why access reviews and non-human identity (NHI) credential rotation are two of the controls that most often pass design and fail operation. It is written for control owners, GRC teams, internal auditors and security leads preparing for SOC 2, ISO/IEC 27001, SOX or NIST-based assessments.

Key takeaways

Two different questions. Design asks "would this control work if run as described?"; operating effectiveness asks "did it run that way, every time, for the whole period?". A control must pass both to be relied on.
Inquiry is never enough. PCAOB AS 2201 ranks evidence from inquiry, through observation and inspection, to re-performance, and states that inquiry alone does not support a conclusion on effectiveness.
Type 1 versus Type 2 is the same split. A SOC 2 Type 1 report covers design as of a point in time; a Type 2 adds operating effectiveness throughout a period, with the auditor's tests and results described.
Frequency drives sample size. The more often a control runs and the higher its risk, the more instances an auditor selects; fully automated controls can rely on a small test plus evidence that change controls kept them unchanged.
Access reviews and NHI rotation fail in operation. The policy and the tool exist, but reviews are rubber-stamped, service accounts are left out of scope and rotation skips credentials believed to be unused.

The two tests side by side

The clearest definitions come from financial audit, where the distinction was formalised. PCAOB AS 2201 tells the auditor to test design effectiveness by determining whether the controls, if operated as prescribed by competent people, satisfy the control objectives and can prevent or detect errors or fraud. It tells the auditor to test operating effectiveness by determining whether the control is operating as designed and whether the person performing it has the necessary authority and competence. Security assurance has adopted the same logic: the AICPA SOC 2 guide describes a type 1 examination as covering whether controls were suitably designed as of a point in time, and a type 2 examination as also covering whether controls operated effectively throughout a period. NIST SP 800-53A Rev. 5 frames the same question through assessment objects (specifications, mechanisms, activities and individuals) and three methods.

Dimension	Test of design (ToD)	Test of operating effectiveness (ToE)
Question	If performed as described, does the control address the risk and meet the objective?	Was the control performed as designed, consistently, throughout the period?
Point in time	As of a date (SOC 2 Type 1, ISO stage 1 style document review)	Across a period, often 3 to 12 months (SOC 2 Type 2, SOX annual testing)
Typical procedures	Walkthrough: inquiry, inspection of the procedure and one end-to-end example, observation	Inspection of evidence for a sample of occurrences, re-performance, system queries
Main evidence	Policy, procedure, RACI, system configuration, control narrative, one traced instance	Dated, attributable records for each sampled instance: tickets, sign-offs, logs, exports
Who is tested	The control as a concept: owner, trigger, precision, inputs	The people and systems that actually ran it, including authority and competence
Failure looks like	Control missing, wrong frequency, imprecise review threshold, input data not complete	Missed instances, late performance, no evidence, performer without authority, exceptions not followed up
Deficiency type	Deficiency in design	Deficiency in operation
Sequence	Always first: a badly designed control is not tested further	Only after design passes

The sequence matters. If the walkthrough shows that a quarterly privileged access review only covers human accounts in the identity provider, the control has a design gap for service accounts and API keys, and sampling twelve months of reviews will not fix that. Teams that skip their own design review often spend a full audit period generating evidence for a control that was never going to pass. For the broader method of judging whether controls work outside an audit, see the Measuring security control effectiveness guide.

Test methods and what each one proves

AS 2201 paragraph .50 orders tests of controls from least to most evidence: inquiry, observation, inspection of relevant documentation, and re-performance. NIST SP 800-53A uses examine (reviewing, inspecting, observing, studying or analysing objects), interview, and test (exercising mechanisms or activities under specified conditions and comparing actual with expected behaviour). The two vocabularies map closely.

Method (AS 2201 / SP 800-53A)	What it proves	What it does not prove	Security example
Inquiry / Interview	The owner understands the control and can describe it	That anything happened; never sufficient alone	Asking the IAM lead how leavers are disabled
Observation / Examine (observe)	The control ran correctly while watched	That it runs the same way when nobody is watching, or ran earlier in the period	Watching an engineer approve a production change in the pipeline
Inspection / Examine (inspect)	A dated, attributable record exists for the sampled instance	That the review behind the record was meaningful (a signature is not a review)	Reviewing a signed access certification export with reviewer comments
Re-performance / Test	The control produces the correct result when the auditor runs it independently	Behaviour outside the tested population or after later changes	Re-running the stale-account query and comparing results with the reviewer's list
System query (full population)	Every instance in the period met the rule, not just a sample	That the query logic and source data were complete	Exporting all secrets with last-rotated dates from the vault and checking each against the policy maximum

Two refinements from the standards are worth knowing. NIST SP 800-53A describes depth and coverage attributes, each with the values basic, focused and comprehensive, set in the assessment plan: higher assurance needs call for more rigour and more breadth. AS 2201 paragraph .61 asks auditors to vary the nature, timing and extent of tests from year to year to introduce unpredictability, so the evidence that passed last year is not a guaranteed template for this year.

Sampling: how auditors choose what to look at

For manual controls the auditor selects instances from the full population of occurrences in the period. The size of the selection is set by the audit firm's methodology, driven by how often the control runs, how much risk it addresses and whether deviations are expected. AS 2201 paragraph .19 states the general principle: as the risk associated with a control increases, the auditor needs to do more of their own work. The population arithmetic below is fixed; the sample size per frequency comes from your auditor, so ask for their table at planning and write it into your evidence calendar.

Control frequency	Occurrences in a 12-month period	Implication for evidence	Security example
Annual	1	Every instance is tested; one miss is a full failure	Annual review of the access control policy
Quarterly	4	Usually most or all quarters selected; a late quarter is visible immediately	Quarterly privileged and NHI access review
Monthly	12	Several months selected; gaps in evidence retention show quickly	Monthly review of vault audit logs for unused secrets
Weekly	52	Sampled; the auditor picks dates, so every week needs evidence	Weekly review of new OAuth grants in the tenant
Daily or many times a day	365 or more	Sampled from the full population list, which you must be able to produce	Change approvals before production deployment
Fully automated	Continuous	Small test of the configured rule plus evidence that change management kept it unchanged	Vault policy that rotates a database credential every 24 hours

Automated controls get lighter treatment for a reason. AS 2201 Appendix B states that entirely automated application controls are generally not subject to breakdowns due to human failure, and allows a benchmarking strategy where general controls over program changes, access and operations remain effective and the auditor verifies the automated control has not changed since the baseline. The practical lesson for security teams: moving a manual control (an engineer rotating keys from a runbook) into a configured, change-controlled mechanism (rotation enforced by the secrets platform) reduces both the risk of failure and the volume of evidence you must produce. The condition is that the IT general controls around that platform hold, especially access to change its policies.

Producing a population the auditor will accept

Completeness: the auditor will test whether the list you provide contains every occurrence. A list of access reviews that omits the cloud tenant nobody remembered is a scope finding, not a sampling detail.
Accuracy: system-generated reports used as audit evidence (often called information produced by the entity) need their own support, such as the query text, the run date and who ran it.

Evidence that passes and evidence that fails

The most common reason an operating test fails is not that the control did not happen, but that nobody can prove it did. Good evidence is dated, attributable to a named performer, shows the input the performer saw, records the decision and links to follow-up. Use the Audit Evidence Request Tracker Template to manage requests across a period.

Control	Evidence that passes	Evidence that fails
Quarterly user access review	Export of the reviewed entitlement list with run date, reviewer decisions per line, revocation tickets closed with timestamps	An email saying "reviewed, all fine"; a spreadsheet with no date or reviewer; revocations never actioned
Service account and NHI review	Inventory of service accounts, API keys, tokens and OAuth apps with owner, last use and decision per item	A review that only covered accounts in the human identity provider
Credential rotation	Vault or key-management report showing last-rotated date per secret against the policy maximum, plus exception register	A runbook and a statement that rotation happens; rotation logs for only the systems that were easy
Joiner-mover-leaver	HR termination list matched to account disable timestamps, with the agreed time limit met	Disable timestamps without the HR source; leavers missing from the sample population
Change approval	Pull request or change record with approver distinct from author, before the deployment timestamp	Approvals dated after deployment; self-approval by the author

Deficiency classes and how severity is judged

AS 2201 Appendix A defines the vocabulary most assurance work borrows. A deficiency in design exists when a control necessary to meet the objective is missing, or an existing control is designed so that the objective would not be met even if it operated as designed. A deficiency in operation exists when a properly designed control does not operate as designed, or when the person performing it lacks the necessary authority or competence. Severity is then graded: a significant deficiency is less severe than a material weakness yet important enough to merit attention by those responsible for oversight, and a material weakness creates a reasonable possibility that a material misstatement will not be prevented or detected on a timely basis.

Class	Typical trigger in security controls	Usual consequence
Exception or deviation	One sampled instance late or missing evidence	Reported in a SOC 2 Type 2 test results section; auditor considers whether it is isolated
Deficiency in operation	Several missed review cycles; reviewer without authority to revoke	Management letter point or qualified result for that criterion, depending on pervasiveness
Deficiency in design	No control over NHI credentials at all; review scope excludes privileged service accounts	Control cannot be relied on; operating testing not performed
Significant deficiency (SOX)	IT general control failure over a financially relevant system, mitigated by other controls	Reported to the audit committee
Material weakness (SOX)	Pervasive access control failure over financial systems with no compensating control	Disclosed; adverse opinion on internal control over financial reporting

In SOC 2 work the outcome language differs: the AICPA guide notes that description misstatements, deviations and deficiencies are all types of misstatement, and a type 2 report includes a description of the auditor's tests of controls and their results, so every exception becomes visible to customers reading the report. See How to Read a SOC 2 Report Guide for how readers interpret exceptions and modified opinions.

Worked failures: access reviews and NHI credential rotation

Access reviews

Design looks fine: a quarterly certification campaign in the identity governance tool, managers as reviewers, revocation workflow attached. Operation fails because: reviewers approve hundreds of lines in seconds, the campaign excludes accounts in SaaS applications not connected to the tool, and revocation tickets are raised but not closed before the next quarter. An auditor re-performing the review will pick a sampled user, compare current entitlements with role need and find access the reviewer approved. Fix the design (scope, precision, segregation of duties between reviewer and requester) before the period starts, then make operation measurable. The Access Reviews Guide and Access Review Checklist cover campaign design in depth.

NHI credential rotation

Design looks fine: a policy that every service account password, API key and token rotates at a defined maximum age, with the secrets platform as the system of record. Operation fails because: the vault holds only the credentials that were migrated, hard-coded keys in pipelines never appear in the rotation report, and rotation after an incident skips credentials believed to be unused. The Cloudflare breach shows the cost: after the Okta breach, one service token and three service account credentials were not rotated because they were mistakenly believed to be unused, and the attacker used them. An auditor testing rotation against the vault's own report would have passed the control; testing against a complete NHI inventory would not. See the Credential Rotation at Scale guide and How to Run an Access Review for Service Accounts.

Where identity fits

Most security control populations are identity populations: accounts, entitlements, credentials, approvals. That is why NHIs create audit problems. Machine identities outnumber people, have no manager to certify them, and often sit outside the HR-driven joiner-mover-leaver process that auditors understand. When you scope a control, ask whether the population includes service accounts, workload identities, API keys, OAuth applications and AI agents, and who the accountable owner is for each. The Audit Readiness for AI Agents and NHIs guide lists the evidence auditors ask for on these populations.

How to prepare: a phased approach

Write precise control narratives. For each key control state the objective, trigger, frequency, performer, population, precision (what counts as an exception) and evidence produced. Vague narratives are the main source of design deficiencies.
Run your own walkthrough. Trace one instance end to end before the period starts, exactly as an auditor would. Fix design gaps now: once the period begins, a redesign means a new period.
Prove the population. Build the query or report that lists every occurrence (every review, every leaver, every secret) and document its logic, source and owner.
Automate where you can. Move high-frequency manual steps into configured, change-controlled mechanisms, and make sure access to change those mechanisms is itself controlled and logged.
Pilot an internal operating test. Mid-period, have internal audit or a second line team select a sample and inspect evidence. Treat any exception as a real one: root cause, remediation, and whether it is isolated.
Manage exceptions openly. Record each deviation, its cause and the compensating evidence. Auditors treat a documented, remediated exception very differently from one they discover.
Roll back cleanly when a control changes. If you replace a manual control with an automated one mid-period, keep evidence for both segments and tell the auditor the change date; an unexplained switch looks like a gap.

Decision guidance

Type 1 or Type 2 first: a Type 1 is useful when controls are new and customers need something quickly, but it says nothing about operation. Plan the Type 2 period to start only once your own walkthroughs pass.
Manual or automated control: choose automation for high-frequency, rule-based controls (rotation, disablement, configuration baselines); keep human judgement for reviews of appropriateness, and make it precise.
Sample or full population: where a system can produce the full population (every secret's age, every account's last sign-in), test all of it internally even if the auditor samples. Full-population testing finds the exceptions a sample misses.
Small versus large organisations: a small company may have one quarterly review and a handful of service accounts, so every instance is tested and a single miss is visible. Large organisations face population completeness problems instead: many tenants, many tools, and control owners who do not know every system in scope.

Detection and measurement

On-time completion rate: percentage of control instances completed by their due date in the period (target close to 100 per cent for annual and quarterly controls).
Evidence completeness: percentage of instances with dated, attributable evidence retrievable within one working day.
Internal exception rate: exceptions found by internal operating tests per control, trending down across periods.
Population coverage: share of known NHIs (service accounts, keys, tokens, OAuth apps) inside the review and rotation scope.
Revocation closure time: median days from review decision to access actually removed.

Common failure patterns

Testing operation before design: a period of evidence collected for a control whose scope excluded the riskiest accounts.
Relying on statements: control owners offer inquiry ("we always do this") instead of records, which AS 2201 says is never sufficient alone.
Rubber-stamp reviews: certification evidence exists, but re-performance shows inappropriate access was approved.
Incomplete populations: rotation and review reports cover only what is in the tool, so hard-coded and unmanaged NHI credentials never appear, as in the Internet Archive breach where unrotated Zendesk keys enabled a second breach.
Unprotected automation: an automated control is relied on while access to change its configuration is uncontrolled, which breaks the benchmarking condition.

Framework mapping

Framework	Reference	Relevance
NIST SP 800-53 Rev. 5	CA-2 Control Assessments; CA-7 Continuous Monitoring	Requires assessing whether controls are implemented correctly, operating as intended and producing the desired outcome
NIST SP 800-53 Rev. 5	AC-2 Account Management; IA-5 Authenticator Management	The controls behind access reviews and credential rotation
NIST CSF 2.0	GV.OV-03; ID.IM-02	Performance of risk management is evaluated; improvements identified from tests
ISO/IEC 27001:2022	Clause 9.2 Internal audit; Annex A 5.35 and 5.36	Independent review and compliance checking of controls
AICPA SOC 2	Type 1 and Type 2 examinations	Design as of a date versus operation throughout a period
PCAOB AS 2201	Paragraphs .42 to .45, .50, Appendix A and B	Design and operating tests, evidence hierarchy, deficiency definitions, benchmarking

Practitioner checklist

Write each key control narrative with objective, frequency, performer, population, precision and evidence.
Walk through one instance of every key control before the audit period starts.
Confirm review and rotation scopes include service accounts, API keys, tokens, OAuth apps and AI agents.
Document the query or report that produces each control population, with its source and logic.
Ask the auditor for their sample size table and build an evidence calendar from it.
Run an internal operating test mid-period and treat its exceptions as real findings.
Protect automated controls with change management and restricted configuration access.
Track on-time completion, evidence completeness, NHI coverage and revocation closure time monthly.

Control Design vs Operating Effectiveness: What Auditors Actually Test