Most organisations run a shadow AI discovery exercise once, publish an AI policy and then lose sight of unsanctioned AI use again within a quarter, because new AI tools, browser extensions and OAuth-connected apps appear every week. This health check is the recurring measurement that stops that drift: eight metrics with formulas, data source types and green, amber and red thresholds, plus a cadence, a findings log and a remediation map. It is written for the security, data protection or AI governance lead who owns AI use, and for internal audit when it re-performs the figures. How to build discovery, how to choose between blocking and sanctioning, and how to score maturity are covered elsewhere and linked, not repeated here.
What this health check covers
The Shadow AI & AI Use Maturity Model tells you which level your programme has reached; the Network vs Browser vs Endpoint Shadow AI Discovery guide explains how to collect the raw signal. This page sits between them: it turns discovery output into eight numbers you track every month and quarter, so you can see whether unsanctioned AI use is shrinking, whether sanctioned tools are winning, and whether the controls you rely on are still switched on. It suits anyone searching for a shadow AI audit, a way to measure shadow AI usage, or GenAI usage KPIs with targets.
The metrics cover three channels of unsanctioned AI: web AI apps used from a browser, AI apps connected to corporate accounts through OAuth consent, and AI browser extensions. AI agents and MCP servers built by staff are a fourth channel; where you already track them, add them to the S1 denominator and see Shadow AI vs Shadow MCP.
Key takeaways
- Measure against discovery, not declarations: every metric uses network, browser, identity provider or endpoint data as its source, because self-declared AI use only reproduces the register you already have.
- Sanctioned adoption is the lead metric: a falling count of unsanctioned apps means little if sanctioned alternative adoption is flat, since staff may simply have moved to personal devices.
- OAuth grants and extensions are the quiet channel: they keep standing access to corporate data after the user forgets them, so S5 and S8 are measured every month, not every quarter.
- Blocks are not success: blocked sensitive uploads (S3) are read alongside the trend in attempts and in coaching outcomes, never as a target to increase.
- Thresholds are starting points: the values below are NHI Mgmt Group recommendations; record your own risk appetite and the reason for any change in the findings log.
How to use this template
Copy the metrics table and findings log into your own tracker. Fix the definitions once (what counts as an "AI app", which data classes are "sensitive", which tools are sanctioned) and write them into the Shadow AI Policy Template or your AI acceptable use policy so that results are comparable between periods. Collect each figure from the data sources listed, score it, and log every amber or red result as a finding with an owner and due date. Keep each completed run: the quarter on quarter trend tells you more than any single score.
Cadence and sign-off
| Review | Frequency | Owner | Approver | Evidence kept |
|---|---|---|---|---|
| OAuth grant and extension sweep (S5, S8) | Monthly | Identity or SaaS security team | Shadow AI programme owner | Grant and extension exports, revocation tickets |
| Discovery reconciliation (S1, S2, S3) | Monthly | Security operations or data protection team | Shadow AI programme owner | Discovery exports, AI app catalogue snapshot, DLP event summary |
| Full metric run (S1 to S8) | Quarterly | Shadow AI programme owner | CISO or chair of the AI governance committee | Metric workbook, findings log, committee minutes |
| Independent re-performance | Yearly | Internal audit | Audit committee | Re-performed samples, variance notes |
Metrics
"AI app" means any service whose main function is generating or transforming content with a model, including AI features you have not approved inside otherwise sanctioned SaaS. "Unsanctioned" means not on the approved AI app catalogue for the data class involved. NHI Mgmt Group recommended thresholds; adjust to your risk appetite.
| ID | Metric | Formula (numerator / denominator) | Data source type | Green | Amber | Red |
|---|---|---|---|---|---|---|
| S1 | Unsanctioned AI app count | Distinct unsanctioned AI apps with at least one corporate user in the period; trend against the previous period | Secure web gateway or SSE logs; CASB app catalogue; browser telemetry; DNS logs | Falling or flat, every app triaged within 30 days | Rising, or untriaged apps 30 to 60 days old | Rising for two periods, or untriaged apps over 60 days |
| S2 | Users on unsanctioned AI | Distinct users with any unsanctioned AI activity / users with any AI activity | Same as S1, joined to identity provider user IDs | < 10% | 10% to 25% | > 25% |
| S3 | Sensitive data uploads blocked | Sensitive data events to AI apps blocked or redirected / sensitive data events to AI apps detected; plus events per 1,000 AI sessions (trend) | DLP (browser, endpoint or inline) policy events tagged to the AI category | ≥ 95% handled, rate falling | 85% to below 95%, or rate flat | < 85%, or rate rising for two periods |
| S4 | Sanctioned alternative adoption | AI sessions on sanctioned tools / all AI sessions | Gateway and SSE logs; licence usage reports from sanctioned tools | ≥ 80% | 60% to below 80% | < 60% |
| S5 | AI OAuth grants | OAuth grants to AI apps that are unreviewed or carry broad scopes (mail, files, calendar read and write) / all OAuth grants to AI apps | Identity provider and SaaS admin consent and app grant inventories | 0% unreviewed broad grants | Any unreviewed broad grant under 30 days old | Any unreviewed broad grant 30 days or older, or a grant to an app with a known vendor compromise |
| S6 | Policy acknowledgement | In-scope staff who acknowledged the current AI use policy version / in-scope staff | Learning management or policy attestation records; HR role data | ≥ 95% | 80% to below 95% | < 80% |
| S7 | Exception age | Open AI use exceptions past their expiry date / open AI use exceptions; plus median age of open exceptions | Exception or risk acceptance register | 0% past expiry | Any past expiry under 30 days | Any past expiry 30 days or more, or any exception with no expiry date |
| S8 | Browser extension AI count | Installed AI extensions not on the extension allowlist / managed browsers; plus distinct unapproved AI extensions | Browser management console extension inventory; endpoint management software inventory | 0 unapproved extensions with page-read permissions | Unapproved extensions present but triaged within 30 days | Untriaged extensions with access to all sites, or extensions on unmanaged browsers used for work |
Notes on the formulas
- S1 counts apps, S2 counts people: a single popular tool can drive S2 red while S1 is green. Read them together and prioritise by data class, not by count.
- S3 needs both a handled rate and an attempt rate: a high block rate with rising attempts means demand is unmet. Coaching prompts and redirection to a sanctioned tool count as handled; silent logging does not. The Block vs Sanction vs Coach guide covers the choice.
- S4 is the outcome metric: it is the one to report upwards. If S4 rises while S1 falls, the programme is working. If both fall, check S2 and discovery coverage before claiming success.
- S5 scope matters more than count: a grant with offline access to mail or files is standing access held by a third party. The Vercel Context.ai OAuth Breach began with a work account granted broad access to a third-party AI app; the grant stayed valid after the AI vendor was compromised.
- S6 tracks the current version: an acknowledgement of last year's policy does not count once the policy changes. Role-based AI literacy is measured separately in your training metrics; see the AI Literacy Training Plan Template.
- S8 weights by permission: an AI sidebar extension that can read every page sees the same data as the user. Count extensions with all-sites or page-read permissions separately. See the AI Browser Extensions Security Guide.
How to collect the data
- Freeze the definitions: snapshot the approved AI app catalogue, the extension allowlist and the sensitive data classes as at the run date, so later catalogue changes do not rewrite past results.
- Export discovery data: pull AI category traffic from the web gateway, SSE or CASB, and from browser telemetry where deployed. Join to identity provider user IDs and remove service and test accounts for S1, S2 and S4.
- Pull DLP events: filter policy events to the AI destination category and split by action (block, redirect, coach, log) for S3.
- Export OAuth grants: list every third-party app consent in the identity provider and in the main SaaS suites, classify which apps are AI, and record scopes, grant date, granting user and last use for S5.
- Pull attestation and exception records: policy acknowledgements joined to HR role data for S6; the exception register with expiry dates for S7.
- Export extension inventories: from the browser management console and endpoint software inventory, matched against the allowlist, with permissions, for S8.
- Sample and verify: trace five users and five AI apps end to end through the data. A metric that cannot be traced to source records is reported amber at best.
- Compare with the last run: record each metric's movement and note any change in discovery coverage (new sites, new browsers, a gateway bypass) that explains it.
Reading the results together
| Pattern | Likely meaning | First action |
|---|---|---|
| S1 falling, S4 rising | Sanctioned tools meet demand | Keep the cadence; review sanctioned tool licences and data settings |
| S1 falling, S4 flat, S2 falling | Use moved out of view (personal devices, mobile data) | Check discovery coverage and S8; survey users on what they need |
| S3 attempts rising, block rate high | Unmet demand for a capability | Fast-track a sanctioned option for the use case behind the attempts |
| S5 red with S1 green | Risk sits in standing grants, not live traffic | Revoke unreviewed broad grants; move AI apps to admin consent |
| S7 rising | Approval process is too slow, so exceptions become the norm | Measure request decision time and publish a target |
Red flags that need action now
- An OAuth grant with mail or file scopes to an AI app whose vendor has disclosed a compromise (S5).
- Source code, customer data or credentials detected going to an unsanctioned AI app with only a log action (S3). The Samsung ChatGPT Source Code Leak shows how quickly this happens once a tool is popular.
- An unapproved AI extension with access to all sites installed on administrators' browsers (S8).
- Exceptions with no expiry date, or expired exceptions still in use (S7).
- A sudden fall in all AI traffic with no programme change, which usually means a discovery source has broken.
- Sanctioned tool adoption below half of AI sessions for two quarters running (S4).
Findings log
| Finding ID | Finding | Metric | Severity | Owner | Due date | Status |
|---|---|---|---|---|---|---|
| HC-SAI-01 | [Number] unreviewed OAuth grants with broad scopes to AI apps | S5 | [High / Medium / Low] | [Role] | [Date] | [Open / In progress / Closed / Risk accepted] |
| HC-SAI-02 | [Number] unapproved AI extensions with all-sites permission | S8 | [Severity] | [Role] | [Date] | [Status] |
| HC-SAI-03 | Sanctioned adoption at [%], below target | S4 | [Severity] | [Role] | [Date] | [Status] |
| HC-SAI-04 | [Number] expired AI use exceptions still active | S7 | [Severity] | [Role] | [Date] | [Status] |
| HC-SAI-05 | [Finding] | [Metric] | [Severity] | [Role] | [Date] | [Status] |
Group items with a shared root cause into one finding. Risk accepted findings carry an expiry date and the approver's name, and are re-opened at the next quarterly run.
Where identity fits
Two of the eight metrics are identity metrics in disguise. Every AI OAuth grant (S5) is a non-human identity: a token held by a third party, acting with a user's delegated permissions, often with a refresh token that outlives the user's interest in the tool. Treat AI app grants like any other third-party integration, with an owner, scope review and expiry, as set out in the Consent Phishing and Illicit OAuth Grants Guide. S2 also depends on identity: it is only accurate when traffic is attributed to a named corporate account, so unauthenticated proxy traffic or shared accounts weaken it.
Framework mapping
| Framework | Reference | Metrics |
|---|---|---|
| NIST CSF 2.0 | ID.AM-02 Inventories of software, services and systems; ID.AM-04 Inventories of supplier services | S1, S5, S8 |
| NIST CSF 2.0 | GV.PO-01 Policy established and communicated; PR.AT-01 Awareness and training | S6, S7 |
| NIST CSF 2.0 | DE.CM-09 Hardware, software and data monitored | S2, S3, S4 |
| NIST SP 800-53 Rev. 5 | CM-8 System Component Inventory; CM-10 Software Usage Restrictions; CM-11 User-installed Software | S1, S8 |
| NIST SP 800-53 Rev. 5 | AC-20 Use of External Systems; PL-4 Rules of Behavior; PM-6 Measures of Performance | S2, S6, all |
| NIST AI RMF | GOVERN 1.6 AI system inventory; GOVERN 6.1 Third-party AI risk | S1, S5 |
| CIS Controls v8.1 | Control 2 Inventory and Control of Software Assets | S1, S8 |
| MITRE ATT&CK | T1567 Exfiltration Over Web Service; T1528 Steal Application Access Token | S3, S5 |
Remediation map
| Finding type | Metrics | Where the fix is explained |
|---|---|---|
| Discovery gaps | S1, S2 | Network vs Browser vs Endpoint Shadow AI Discovery; Shadow AI Discovery Guide |
| Data leaving to AI apps | S3 | GenAI DLP vs Traditional DLP; Browser DLP vs Endpoint DLP |
| Low sanctioned adoption | S4 | Enterprise AI Licences vs Consumer AI Tools; AI Gateway vs CASB/SSE for Controlling Shadow AI |
| Risky OAuth grants | S5 | Consent Phishing and Illicit OAuth Grants Guide |
| Policy and exception failures | S6, S7 | Shadow AI Policy Template; AI Acceptable Use Policy Template |
| Unapproved extensions | S8 | AI Browser Extensions Security Guide; Browser Extension Security Guide |
Related guides
- Shadow AI & AI Use Maturity Model: place these results on a five-level scale.
- Top 10 Shadow AI Risks: the risks these metrics track.
- Shadow AI Statistics: external benchmarks to compare your S2 and S4 results with.
- Shadow AI Executive Briefing: how to present the programme to leadership.
- AI Governance Programme Health Check: the wider AI governance metrics this page feeds.
References
- NIST: AI RMF Playbook, GOVERN function
- NIST: Cybersecurity Framework 2.0
- NIST: SP 800-53 Rev. 5
- CIS: Control 2, Inventory and Control of Software Assets
- ICO: Artificial intelligence guidance
Related NHI Mgmt Group resources: Shadow AI & AI Use Maturity Model, Network vs Browser vs Endpoint Shadow AI Discovery, Shadow AI Policy Template, Shadow AI Statistics.