Identity controls are usually tested once, when they are built, and then assumed to keep working while group memberships, app registrations, exclusions and role assignments change underneath them. Continuous identity control validation replaces that assumption with tests: for each preventive control, state the outcome it must produce, then check it on a schedule and after every change. This guide covers how to write those expected outcomes, how to test conditional access without real sign-ins, how to find MFA enforcement gaps, how to prove privileged and just-in-time access actually expires, and how to report the results as assurance evidence. It is written for identity engineers, security assurance teams and auditors. Detection testing for identity attacks is in ITDR validation, and endpoint and network control simulation is in the Breach and Attack Simulation (BAS) Guide; this page covers the preventive identity layer that neither tests directly.
Key takeaways
- Write the outcome, not the setting. "Every interactive sign-in by a privileged role from outside a compliant device is blocked" is testable; "CA policy 12 is on" is not.
- Simulate first, observe second. Policy simulators answer "would this be enforced"; sign-in and activation logs answer "was it enforced". Assurance needs both, because simulators skip conditions they are not given.
- Drift is the main failure mode. Exclusions added during incidents, new apps outside policy scope and permanent role assignments created "temporarily" are what break controls, not the original design.
- Coverage gaps hide in definitions. A user who is "MFA capable" may still sign in with a single factor if no policy requires MFA for that app, client or location.
- Every failed assertion is a ticket. A result with no owner and no due date is a finding, not assurance.
Validation, posture management and detection testing
Three practices look similar and are often bought or staffed as one. Keeping them distinct avoids gaps.
| Practice | Question it answers | Typical evidence | Where covered |
|---|---|---|---|
| Identity control validation (this guide) | Does each preventive control produce the outcome we require, today, for every identity in scope? | Pass or fail per assertion, simulator results, sign-in log samples | This page |
| Identity security posture management | Which misconfigurations and risky identities exist? | Findings list, risk scores | ISPM Guide |
| Detection validation | When an identity attack happens, does an alert fire and reach someone? | Detection rate per technique, time to alert | ITDR validation |
| Breach and attack simulation | Do endpoint, network and email controls stop emulated techniques? | Prevented, detected, missed per scenario | Breach and Attack Simulation (BAS) Guide |
NIST SP 800-137 frames the goal: continuous monitoring gives ongoing assurance that controls remain effective and aligned with risk tolerance. In SP 800-53 terms this is CA-7 Continuous Monitoring fed by CA-2 Control Assessments, rather than a once-a-year audit sample.
Writing expected outcomes as assertions
An assertion has four parts: the identity population, the access path, the expected result and the evidence source. Write them from your policy and from an external baseline so you are not only testing what you already built. CISA's SCuBA Microsoft Entra ID baseline is a useful starting set because its policies are already phrased as outcomes:
| Control area | Example assertion | External anchor | Primary test method |
|---|---|---|---|
| Legacy authentication | Any sign-in using a legacy client by any user is blocked | SCuBA MS.AAD.1.1v1 | Simulator with legacy client app; sign-in log query for legacy clients with success |
| Risk-based blocking | High-risk users and high-risk sign-ins are blocked | MS.AAD.2.1v1, MS.AAD.2.3v1 | Simulator with sign-in risk High; review of risky sign-ins that succeeded |
| MFA for everyone | Every user, including guests, must complete MFA for every cloud app | MS.AAD.3.2v2 | Coverage query on sign-ins completed with a single factor |
| Phishing-resistant MFA | Highly privileged roles can only sign in with phishing-resistant methods | MS.AAD.3.6v1 | Simulator per privileged role; sign-in method sample |
| Role population | Between two and eight Global Administrators | MS.AAD.7.1v1 | Directory count, compared each run |
| No standing privilege | No permanent active assignments to highly privileged roles | MS.AAD.7.4v1 | Role assignment inventory |
| Privilege only through PAM | No highly privileged role is assigned outside the PAM system | MS.AAD.7.5v1 | Audit log of role assignments not made by the PAM workflow |
| Activation control | Global Administrator activation requires approval and raises an alert | MS.AAD.7.6v1, MS.AAD.7.8v1 | Role setting read plus a scheduled test activation |
Add assertions the baseline does not cover: service principals and workload identities that should be blocked from unexpected locations, break-glass accounts that must stay excluded from blocking policies but must alert on every use, and guest access to admin portals. Non-human identities often fall outside user-centred policies; see How to Find Accounts Excluded From MFA and Conditional Access and the Break-Glass Account Guide.
Testing conditional access without real sign-ins
Policy simulation is the fastest feedback loop because it needs no test accounts signing in. Microsoft Entra ID's worked example shows both the value and the traps.
- What If tool and API: simulates a sign-in for a user, an agent identity (in preview) or a single tenant service principal against policies that are enabled or in report-only mode. Identity, target resource, device platform and client app are required; other conditions default to none. Results list policies that apply, with the grant and session controls required, and policies that do not apply, with the first unmet condition as the reason. The same engine is exposed as the What If Evaluation API in Microsoft Graph, so assertions can run in a pipeline.
- Complete inputs: Microsoft's example shows a policy scoped to location and high sign-in risk returning "Does not apply" until all of application, location and sign-in risk are supplied. A test that omits conditions passes for the wrong reason, so each assertion should fix every condition the policy uses.
- App groups and dependencies: groups such as Office 365 or Microsoft Admin Portals do not match in the simulator, which needs individual app IDs, and service dependencies are not tested. Test the specific apps your assertion names.
- Report-only before enforcement: report-only policies are evaluated at real sign-ins and logged as Report-only: Success, Failure, User action required or Not applied, without prompting users. User Actions scope cannot be evaluated this way. Microsoft warns that report-only policies requiring a compliant device can repeatedly prompt macOS, iOS and Android users to pick a certificate, so exclude those platforms from report-only compliance policies.
Other platforms offer comparable simulation for authorisation policy. The AWS IAM policy simulator evaluates identity policies, permissions boundaries, service control policies and supplied resource policies and returns allowed or denied per action, without making a real request. AWS notes results can differ from live behaviour for VPC endpoint policies, role chaining or multiple resource-based policies, and that resource control policies are not supported. Whatever the platform, treat simulator output as a fast pre-check and confirm with observed outcomes. See the Conditional Access Guide for policy design.
Detecting policy drift
Drift validation compares today's control state with the last approved state and re-runs assertions whenever it changes.
- Snapshot policy as code: export conditional access policies, authentication method policies and PIM role settings (Microsoft Graph exposes PIM rules through
unifiedRoleManagementPolicy) to version control on each run. - Diff and classify: flag new exclusions, state changes from On to Report-only or Off, removed grant controls, narrowed app scope and new named locations. Exclusions are the highest-risk change because they are invisible in a policy's headline.
- Re-run affected assertions: a membership change to an excluded group should trigger every assertion that group touches, not wait for the weekly run.
- Baseline scanners on a schedule: CISA's ScubaGear queries Microsoft 365 APIs with PowerShell and evaluates the settings against Rego policies written from the SCuBA baselines using Open Policy Agent, producing HTML, JSON and CSV. The open source Maester framework runs Pester-based PowerShell tests and emits HTML, JSON and JUnit output for pipelines, and skips checks with a reason code when a licence or service is missing, which you should report rather than silently accept.
Broader configuration drift practice is in the Configuration Drift and Baseline Enforcement Guide.
Finding MFA enforcement gaps
MFA coverage has two halves that are often confused: registration (can the user do MFA) and enforcement (was MFA required on this sign-in). Microsoft's registration details report defines "MFA capable" as registered for a strong method that policy also allows for MFA; it lags by up to 36 hours and omits disabled and soft-deleted users. A user can be MFA capable and still sign in with a password alone if no policy requires MFA for that path.
- Enforcement test: query successful sign-ins that completed with a single factor, grouped by application, client type and policy result. Each group is either a documented exception or a gap.
- Path inventory: list every way to authenticate (IdP, legacy protocols, VPN and remote access portals, local application accounts, API keys) and assert MFA or an approved alternative for each. The Change Healthcare breach began with a stolen credential on a remote access portal that did not require MFA.
- Forgotten tenants and test environments: include non-production tenants in scope. The Midnight Blizzard breach started with a password spray against a legacy test tenant account without MFA.
- Method strength: for privileged roles, assert the method used was phishing resistant, not only that MFA happened. See MFA Coverage Measurement for the metric design.
Proving privileged and just-in-time access expires
JIT controls fail silently: an activation that lasts longer than intended or an eligible assignment that never ends looks normal in day to day use. Test the settings and the behaviour.
| Assertion | Setting to read (Entra PIM example) | Behavioural test |
|---|---|---|
| Activations end within the approved window | Activation maximum duration (1 to 24 hours) | Activate a test role and confirm the assignment is removed at expiry |
| Activation requires strong authentication | On activation, require MFA, or require a Conditional Access authentication context | Activate from a session without fresh strong authentication and confirm a prompt |
| High-impact roles need a second person | Require approval to activate, with named approvers | Submit a test activation and confirm it waits for approval |
| Eligibility itself ends | Expire eligible assignment after, rather than Allow permanent eligible assignment | Report eligible assignments with no end date |
| No standing active privilege | Expire active assignment after, rather than Allow permanent active assignment | Report active assignments to privileged roles with no end date |
| Activation is visible | Notifications for activation of privileged roles | Confirm the alert arrives in the monitored channel, not only an inbox |
Two documented limits shape these tests. Microsoft states the ticket number on activation is not validated against any ticketing system, so check it against your ITSM records yourself. Requiring MFA on active assignment applies when an administrator creates the assignment; PIM cannot enforce MFA when that assignment is later used, so the sign-in path still needs its own conditional access assertion. The same pattern applies to cloud PAM and secrets brokers that issue time-bound credentials: assert on issued lifetime and test that the credential stops working. See the JIT Access Guide and the Privileged Identity Management (PIM) Guide.
Implementation roadmap
- Weeks 1 to 2, catalogue: list preventive identity controls, owners and the outcome each must produce. Map them to the SCuBA policies or your own standard.
- Weeks 3 to 4, read-only checks: run baseline scanners and settings reads with a read-only identity. Record the starting pass rate; do not fix anything yet so the baseline is honest.
- Month 2, simulation suite: encode conditional access assertions as simulator calls with every condition supplied, run them in a pipeline on each policy change and nightly.
- Month 3, observed outcomes: add sign-in and activation log queries that confirm what the simulator predicted, and scheduled test activations for JIT roles using dedicated, named test identities with expiry dates.
- Ongoing, change-triggered runs: trigger affected assertions on policy, group and role changes. Failed assertions open tickets with the control owner.
- Rollback and safety: validation jobs must never change policy. If a test identity or job credential is misused, disable it; nothing else should depend on it.
The validation service's own credential is a sensitive non-human identity: it can read every policy and role assignment. Give it read-only scopes, keep it in a secrets store, monitor its sign-ins and review it with other privileged workload identities.
Measuring and reporting assurance
- Assertion pass rate: passing assertions over total, by control area, trended weekly.
- Time to detect drift: hours between a control-weakening change and the failed assertion; target same day.
- Time to remediate: days from failed assertion to pass, by severity.
- Single-factor sign-in rate: successful single-factor sign-ins not covered by a documented exception, trending to zero.
- Standing privilege count: permanent active and permanent eligible assignments to privileged roles.
- Skipped checks: assertions not evaluated because of missing licence, permission or data; these are coverage gaps, not passes.
Report results as evidence for audits: each assertion with its last run, result and linked ticket. For the board, roll up to trend lines on the pass rate and standing privilege.
Common failure patterns
- Testing the policy you wrote, not the paths that exist: assertions only cover applications already in scope, so new apps and legacy portals never get tested.
- Incomplete simulator inputs: omitting location or risk makes a policy "not apply" and the assertion silently passes or fails for the wrong reason.
- Exclusions that outlive the incident: a group excluded during an outage stays excluded and grows; no assertion checks exclusion membership.
- Registration mistaken for enforcement: high "MFA capable" percentages reported as MFA coverage while single-factor sign-ins continue.
- JIT settings checked, expiry never observed: settings look right but nobody has watched an activation end or an eligibility lapse.
- Skipped checks counted as passes: licence-dependent checks that did not run are reported green.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| NIST SP 800-53 Rev 5 | CA-2 Control Assessments; CA-7 Continuous Monitoring; CA-7(4) Risk Monitoring | Scheduled and change-triggered assertion runs as ongoing assessment |
| NIST SP 800-53 Rev 5 | AC-2(2) Automated Temporary and Emergency Account Management; AC-6(7) Review of User Privileges | JIT expiry and standing privilege assertions |
| NIST CSF 2.0 | PR.AA-05; GV.OV-03; ID.IM-02 | Access permissions enforced and reviewed; performance evaluated; improvements from tests |
| CISA SCuBA | MS.AAD.1.1v1, MS.AAD.3.2v2, MS.AAD.3.6v1, MS.AAD.7.4v1, MS.AAD.7.5v1 | Outcome statements used as assertions |
| MITRE ATT&CK v20 | T1078.004 Cloud Accounts; T1556.006 Multi-Factor Authentication; T1098 Account Manipulation | Techniques that exploit or create the gaps these assertions look for |
Practitioner checklist
- Write each preventive identity control as an assertion with population, path, expected result and evidence source.
- Start from the SCuBA Entra ID baseline or your equivalent standard, then add NHI and break-glass assertions.
- Supply every condition a policy uses when calling a simulator, and test named apps rather than app groups.
- Confirm simulator predictions against sign-in logs before treating them as evidence.
- Export identity policy to version control and alert on new exclusions and state changes.
- Measure single-factor sign-ins, not just MFA registration.
- Run scheduled test activations to prove privileged roles expire and require approval.
- Report skipped checks as gaps and give every failed assertion an owner and due date.
- Keep the validation job's identity read-only and monitored.
Standards and references
- NIST: SP 800-137 Information Security Continuous Monitoring
- CISA: SCuBA Microsoft Entra ID Baseline
- CISA: ScubaGear
- Microsoft Learn: Conditional Access What If tool
- Microsoft Learn: Conditional Access report-only mode
- Microsoft Learn: Configure Microsoft Entra role settings in PIM
- Microsoft Learn: Authentication methods activity
- AWS: Testing IAM policies with the IAM policy simulator
Related NHI Mgmt Group resources: Breach and Attack Simulation (BAS) Guide · ITDR validation · ISPM Guide · Conditional Access Guide · JIT Access Guide