What breaks when AI model risk scoring is based only on vendor claims?

Teams lose the ability to verify how a model behaves under attack, which means hidden jailbreak paths, data leakage, or policy failures can reach production unnoticed. Evidence-based scoring matters because it ties approval to observable tests, lineage signals, and repeatable review criteria rather than marketing language.

Why This Matters for Security Teams

Vendor claims often describe how a model was intended to behave, not how it behaves when exposed to prompt injection, adversarial inputs, poisoned retrieval content, or unsafe tool use. For security teams, that gap breaks the approval process because risk scoring becomes a paper exercise instead of a control decision. NIST’s NIST Cybersecurity Framework 2.0 is useful here because it reinforces the need for governance, risk management, and verification before deployment.

The practical issue is that AI systems fail in context. A model can look compliant in a demo and still leak sensitive data, ignore policy boundaries, or produce unsafe actions once it is connected to live prompts, retrieval sources, or downstream automation. That makes claims-based scoring especially dangerous for agentic systems, where an AI agent may have execution authority, tool access, or the ability to influence business workflows. In those environments, a weak score can create a false sense of assurance and allow hidden failure modes into production.

In practice, many security teams encounter model risk only after an incident reveals that the score was based on promises rather than adversarial testing.

How It Works in Practice

Evidence-based AI risk scoring starts with inputs that can be examined, reproduced, and challenged. That usually means combining provenance checks, controlled test results, and operational telemetry. Rather than trusting a vendor’s statement that a model is “safe,” practitioners should ask what was tested, against which threats, with what datasets, and under what deployment conditions. For AI systems, the right question is not whether the model is generally capable, but whether it remains trustworthy under abuse conditions.

Current guidance suggests building scoring around a few concrete control areas:

• Model provenance, including training data lineage, version history, and change approval.
• Red-team and abuse-case testing for prompt injection, jailbreaks, policy bypass, and data extraction.
• Output validation for unsafe, misleading, or non-compliant responses before release or action.
• Monitoring for inference-time anomalies, especially when the model can call tools or access secrets.
• Human review for high-impact use cases, especially where the model affects customers, financial outcomes, or regulated decisions.

For governance alignment, NIST AI Risk Management Framework and MITRE ATLAS both support a more testable approach because they emphasise risk identification, adversarial thinking, and operational monitoring rather than vendor assurance alone. For teams building agentic workflows, OWASP Agentic AI Top 10 is especially relevant where the system can reason, plan, and act through tools.

This guidance tends to break down when the model is embedded in fast-moving CI/CD pipelines without a defined test gate, because teams then promote releases on schedule pressure instead of scored evidence.

Common Variations and Edge Cases

Tighter model review often increases delivery overhead, requiring organisations to balance faster release cycles against stronger assurance. That tradeoff is real, especially for teams that use multiple foundation models, fine-tuned variants, or third-party APIs. There is no universal standard for model risk scoring yet, so the right depth of testing depends on exposure, data sensitivity, and the model’s authority to take actions.

Some environments need extra caution. A low-stakes internal summarisation model may justify lighter validation, while a customer-facing assistant connected to internal knowledge bases, payment flows, or administrative tools needs much stronger evidence. Where retrieval-augmented generation is involved, claims about the base model say little about the risk created by the retrieval layer, document quality, or policy enforcement around generated answers. In those cases, scoring must cover the full system, not just the model.

Another common edge case is the agentic stack. Once a model can call functions, trigger workflows, or interact with secrets, the risk changes from content quality to execution risk. That is where NHI governance becomes relevant, because the model or agent may need scoped credentials, lifecycle control, and strict boundaries on what it can access. Teams should treat vendor claims as input to review, not as the review itself. For broader operational context, NIST Cybersecurity Framework 2.0 remains a useful anchor for governance and continuous improvement.

Best practice is evolving, but the current consensus is clear: if the score cannot be tied to observable tests and repeatable evidence, it should not be treated as a reliable control decision.

Related resources from NHI Mgmt Group

• What usually breaks when vendor risk scoring is done without evidence?
• When does a credit-based AI model create more risk than it reduces?
• What do identity teams get wrong about AI-based risk scoring?
• What breaks when AI model choice is tied to a single vendor platform?