AI model risk scoring is only as strong as its test coverage

At a glance

What this is: This methodology turns AI model behaviour into 0-100 risk scores and A-F grades by combining red teaming, OSINT, and weighted risk pillars.

Why it matters: It matters because model selection, approval, and monitoring now depend on evidence quality, not just vendor claims, and those decisions intersect with AI governance, identity, and access controls.

By the numbers:

• The composite score uses a weighted hierarchy where Risk_Score = Σ(Pillar_Weight × Pillar_Score) and the weights sum to 100%.

👉 Read Noma Security's methodology for AI model risk scoring and red teaming

Context

AI model risk assessment fails when teams rely on static vendor assurances instead of reproducible evidence. In this article, model risk is measured through adversarial testing, open-source intelligence, and weighted scoring, which is the right direction for AI governance but still leaves judgment about operational context to practitioners.

The model also intersects with identity governance because AI systems are not just outputs, they are runtime entities that may hold secrets, invoke tools, or expose personal data. That means security teams need to evaluate model behaviour alongside access boundaries, data handling, and the trust assumptions behind any connected agent or workflow.

Key questions

Q: What breaks when AI model risk scoring is based only on vendor claims?

A: Teams lose the ability to verify how a model behaves under attack, which means hidden jailbreak paths, data leakage, or policy failures can reach production unnoticed. Evidence-based scoring matters because it ties approval to observable tests, lineage signals, and repeatable review criteria rather than marketing language.

Q: Why do AI agents create a governance problem for IAM teams?

A: AI agents create a governance problem because they authenticate and act as autonomous software entities with tool access. If their actions are logged only as application activity, teams lose accountability, context, and revocation clarity. IAM must therefore extend to agent identity, delegated authority, and control-plane audit trails.

Q: How do security teams know if a model risk score is actually useful?

A: A useful score correlates with specific failure modes such as jailbreak susceptibility, privacy leakage, weak provenance, or inconsistent refusals. Security teams should be able to trace the score back to underlying evidence and see whether the same model changes when retested after updates.

Q: Who should be accountable when a model with a good score still causes harm?

A: The accountable owner is the team that approved the model for a specific use case and operating context. A score does not transfer responsibility. Governance should require named ownership for the model, the data it touches, and the controls that constrain its runtime behaviour.

Technical breakdown

How adversarial red teaming produces measurable AI risk signals

The methodology uses automated red teaming to search for prompts that elicit unsafe or policy-breaking behaviour, rather than relying on fixed test cases. That matters because models adapt during testing, so a live search can expose evasions, jailbreak susceptibility, and inconsistent refusal patterns that static checks miss. The output is then converted into indicators that can be aggregated into broader risk pillars. This is closer to security testing than benchmark scoring, because it measures observed behaviour under pressure rather than claimed capability.

Practical implication: validate model risk with adversarial testing that reflects live abuse conditions, not only pre-launch benchmark scores.

Why OSINT matters in AI model governance

Open-source intelligence adds context that red teaming alone cannot provide. Model cards, repository activity, dataset lineage, and license terms can reveal provenance gaps, hidden dependencies, or weak documentation that affect trust and compliance. In governance terms, this extends evaluation beyond the model’s responses to the evidence surrounding how it was built, updated, and published. For enterprise buyers, that is important because many AI risks emerge from lineage opacity or poor disclosure, not only from prompt injection or jailbreaks.

Practical implication: pair behavioural testing with provenance checks so procurement decisions reflect both model performance and supply-chain trust.

How composite risk scoring should be interpreted by security teams

A composite score can simplify decisions, but only if teams understand what the score actually compresses. The weighted hierarchy combines multiple pillars, each with different evidence sources and operational meanings, so a single grade should never replace inspection of the underlying failure modes. Security and privacy weaknesses may matter more than reputation, while transparency issues may affect auditability rather than immediate exploitation risk. The right use of the score is prioritisation, not delegation of judgment.

Practical implication: use the score to triage models for deeper review, then inspect pillar-level results before approval or deployment.

Threat narrative

Attacker objective: The attacker objective is to coerce the model into unsafe, privacy-invasive, or policy-violating behaviour that creates downstream business and compliance harm.

1. Entry occurs through adversarial prompts, jailbreak attempts, or prompt injection strings designed to surface unsafe model behaviour.
2. Escalation happens when repeated probing identifies responses that leak sensitive data, ignore policy constraints, or reveal weak guardrails.
3. Impact is reached when the model produces harmful, misleading, or non-compliant outputs that can be operationalised in production workflows.

NHI Mgmt Group analysis

AI model risk scoring only works when the evidence is adversarial, not declarative. A letter grade built from live testing is more defensible than one built from vendor assurances or static checklists. The article’s methodology is strongest where it ties scores to observable model behaviour under attack conditions, which is the right direction for AI governance. Practitioners should treat any score as a starting point for control validation, not an approval stamp.

Model governance now overlaps with identity governance because AI systems increasingly act like privileged runtime entities. Once a model can see data, call tools, or influence workflows, it is no longer just a content generator. That creates an identity and access problem around what the model can reach, what it can disclose, and how its actions are contained. Security teams should assess AI systems with the same discipline used for privileged service accounts and other high-risk non-human identities.

Transparency risk is not cosmetic, it is an auditability problem. If a model cannot explain its limitations, lineage, or operating boundaries, security and compliance teams cannot reliably assess where trust ends. That makes model cards, provenance, and disclosure controls part of the control surface rather than optional documentation. Practitioners should expect transparency gaps to show up later as governance debt, especially where regulated data or high-impact decisions are involved.

AI governance debt: the hidden accumulation of unresolved model provenance, testing, and disclosure gaps that makes later controls harder to enforce. This article shows how quickly risk scoring can become a governance crutch if teams stop at the composite number. The better pattern is to use the score to surface where evidence is missing, then drive remediation across red teaming, lineage, privacy, and operational monitoring. Practitioners should align approval processes to the underlying evidence, not the headline grade.

Security, privacy, and safety risks are converging into a single operational decision problem. The same model can be safe in a lab, risky in a workflow, and compliant in one context but not another. That means teams need to evaluate deployment context, connected tools, and data sensitivity together. Practitioners should build model review gates that combine AI risk, access control, and data governance before production use.

What this signals

AI governance debt: teams that adopt composite model scores without evidence review will accumulate hidden testing, lineage, and disclosure gaps that surface later as audit failures or incident response friction. The practical move is to tie approval to retesting, provenance checks, and role-based ownership, not to the final letter grade alone.

As AI systems become more connected to data and tools, they need identity controls that look more like privileged access governance than traditional application review. For practitioners, the question is no longer whether a model is accurate enough, but whether its runtime permissions and data reach are bounded enough for the intended business context.

For practitioners

• Define model approval gates around evidence quality Require red team transcripts, lineage data, and OSINT checks before accepting a model score as fit for use. Treat missing provenance or weak documentation as a blocking issue for high-impact deployments.
• Map AI model access to identity controls Treat models that can call tools, read data, or trigger workflows as privileged non-human identities. Limit their permissions, log their actions, and review their access paths with the same rigour used for service accounts.
• Review pillar-level failures before procurement Do not rely on the final A-F grade alone. Inspect which pillar failed, whether it was security, privacy, or transparency, and decide whether the issue is acceptable for the intended use case.
• Build continuous rescoring into model monitoring Reassess scores when models are updated, because version changes can alter behaviour and risk. Use scheduled rescans for models in production and trigger reviews after major prompt, policy, or provider changes.

Key takeaways

• Evidence-based AI model scoring is only meaningful when it is tied to live adversarial testing and provenance checks.
• AI systems that can call tools or access data create identity and access risks that belong in the same governance process as privileged accounts.
• Composite grades are useful for prioritisation, but practitioners still need pillar-level review before approving a model for production use.

Key terms

• Composite risk scoring: Composite risk scoring combines multiple identity signals into one decision, such as lifecycle state, device trust, authenticator strength, and ticket context. It is only as reliable as the quality and completeness of the input feeds that support it.
• Adversarial Red-Teaming: Adversarial red-teaming is the practice of actively trying to make a security model fail before it reaches production. The test uses crafted inputs, boundary probes, and mutation strategies to reveal weaknesses that ordinary accuracy testing will not show.
• Model Provenance: Model provenance is the evidence chain showing where an AI artefact came from, how it was modified, and whether the version in use is the one that was approved. For AI security teams, provenance is the control that turns trust from assumption into verification.
• Governance Debt: The accumulation of unresolved identity control weaknesses created when teams prioritise speed over lifecycle design. In NHI environments, it shows up as accounts with unclear ownership, undocumented purpose, stale credentials, and no reliable retirement path, all of which make later security work harder.

What's in the full report

Noma Security's full article covers the operational detail this post intentionally leaves for the source:

• The full scoring methodology for converting red team and OSINT evidence into weighted risk indicators.
• Appendix-level detail on the six risk pillars and how each indicator contributes to the composite grade.
• Version tracking and continuous assessment mechanics for rescoring models after updates.
• The underlying behaviour set used by the automated red teaming engine to elicit unsafe responses.

👉 The full Noma Security article includes the scoring logic, pillar decomposition, and red teaming detail behind the framework.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security and identity practitioners build the control thinking needed to govern high-risk runtime identities.