Security Ratings vs Vendor Assessments: How to Combine Them by Vendor Tier

Third-party risk teams keep arguing about whether an outside-in security rating or an inside-out vendor assessment is the "real" answer, when each one is blind exactly where the other can see. A rating watches a supplier's internet-facing footprint continuously but cannot see how it runs identity, backups or the tokens it holds for you. A questionnaire and evidence review can see those controls, but only on the day the supplier answers, and only as honestly as it answers. This guide treats the choice as a design decision: what each method can prove, how each fails, and how to combine them by vendor tier so that effort goes where the risk is. It is for TPRM leads, procurement security reviewers and CISOs setting assessment policy. It does not re-explain how rating platforms or standard questionnaires work; for that, read the TPRM and Security Ratings Buyer's Guide and the Security Questionnaires Guide. Sources were checked on 14 November 2026.

Key takeaways

• They answer different questions. A rating asks "does this supplier's perimeter look neglected or changing?"; an assessment asks "does this supplier run the controls that protect our data and access?"
• Evidence depth decides trust, not method. A self-attested questionnaire and a scan-derived grade are both weak evidence; an independent audit report scoped to the service you buy is strong.
• Tier first, then pick the mix. Critical suppliers need both, with evidence verified; low-tier suppliers may need only a rating watch or nothing at all.
• Each method has a signature failure. Ratings produce false positives from mis-attributed assets; questionnaires produce fiction, answers that are true on paper and false in production.
• Combine them as a loop. Use rating changes to trigger targeted reassessment, and use assessment findings to tell you which external signals to watch.

The short answer

Do not choose one. Tier every supplier on inherent risk (data, access, criticality), then apply the mix in the tier table below. Ratings are cheap, continuous and shallow: use them for coverage across the whole supplier base and as an alarm between assessments. Assessments are expensive, periodic and deep: reserve verified ones for suppliers whose failure would hurt you. US banking regulators' 2023 Interagency Guidance on Third-Party Relationships says the same thing in regulatory language: due diligence and monitoring should be commensurate with the risk and complexity of the relationship, with more rigorous oversight for critical activities.

Threat model: what each method is meant to catch

Outside-in ratings

Ratings are good at detecting the conditions behind opportunistic attacks: an internet-facing service left unpatched, an expired certificate, an exposed database port, weak email authentication records, or credentials appearing in leak sources. They also surface sudden change, such as a new exposed service after an acquisition. Their threat model is "the supplier is an easy target from the internet". They are weak against the threats that drive most damaging third-party incidents today: compromise of the supplier's identity systems, theft of integration tokens, insider misuse and weak internal segregation.

Inside-out assessments

Assessments target the supplier's control environment: how it authenticates staff and machines, how it stores secrets and customer tokens, how it isolates tenants, how it develops and deploys code, how fast it notifies you of incidents, and which subcontractors it relies on. Their threat model is "the supplier's internal controls fail and our data or access is exposed through them". They are weak at detecting change after the assessment date and depend on the supplier's honesty and knowledge.

The Salesloft Drift breach shows the gap clearly: OAuth tokens for a chat integration were taken from the supplier's cloud environment after an intrusion through its code repository account, and used to export data from customer Salesforce tenants. Google Threat Intelligence Group reported over 700 potentially impacted organisations. Nothing in that chain was visible from the internet; the questions that matter (how are customer tokens stored, scoped and rotated, who can reach them) are assessment questions, and they still need evidence behind the answer.

Architecture of each approach in words

Ratings programme. Procurement or the TPRM platform supplies a list of supplier domains. The ratings service maps those domains to IP ranges and assets, scans and collects observations, scores them, and pushes alerts when a score drops or a high-severity finding appears. An analyst triages the alert, confirms the asset belongs to the supplier, and either raises it with the supplier or dismisses it. The weakest step is attribution: shared hosting, content delivery networks, reassigned cloud addresses and divested brands produce findings that are not the supplier's.

Assessment programme. An intake form captures what the supplier will do, which data and access it will have, and how critical the service is. That sets a tier. The tier selects the instrument: a standard questionnaire such as SIG or CAIQ, an independent report such as SOC 2 Type 2 or an ISO/IEC 27001 certificate, targeted evidence requests, and for the top tier, a call, a control walkthrough or an audit. Findings become remediation items with dates and, where needed, contract clauses. The weakest step is verification: most programmes accept answers without checking the few that matter.

Evidence depth: a ladder, not a binary

The real comparison is between levels of evidence. Place each input on a ladder and require a minimum rung by tier.

CSA's STAR programme illustrates the difference between rungs 2 and 4: Level 1 is a self-assessment against the Cloud Controls Matrix, while Level 2 adds a third-party audit through ISO/IEC 27001, SOC 2 or C-STAR. For how to read the report itself, see the How to Read a SOC 2 Report Guide; for when to go beyond reports to audit rights, see Right to Audit vs Assurance Reports.

Combining them by vendor tier

Set the tier from inherent risk before any assessment. The Vendor Tiering and Inherent Risk Questionnaire Template gives a ready intake form. Then apply the mix below.

Small organisations with a handful of critical suppliers often get more from rung 4 and 5 evidence on those few than from a ratings subscription across hundreds of low-risk ones. Free public signals (certificate transparency, breach disclosures, the supplier's trust centre) can stand in for a paid rating at low tiers. Large organisations with thousands of suppliers need ratings for coverage, because no team can assess every supplier inside-out; the discipline is to stop ratings drifting up the tiers and replacing evidence for critical suppliers. The 2023 interagency guidance allows banks to use industry utilities, consortiums and joint due diligence efforts, but says the bank must evaluate those efforts against its own needs and remains responsible for managing the relationship.

Failure modes and how to control them

Rating false positives and false negatives

• Mis-attributed assets: a supplier is marked down for a server on shared hosting it does not own. Control: require analysts to confirm ownership before raising a finding, and use the rating provider's dispute route. The US Chamber of Commerce Principles for Fair and Accurate Security Ratings expect rated organisations to be able to challenge ratings and have disputed ratings marked as disputed.
• Clean score, weak supplier: a small perimeter yields a high grade while identity controls are poor. Control: never let a rating lower a tier or replace evidence for critical suppliers.
• Score as contract weapon: using a grade threshold as a termination trigger invites disputes over methodology. Control: tie contract obligations to specific findings and remediation times, not a number.

Questionnaire fiction

• Aspirational answers: "MFA on all remote access: yes" while one legacy portal is exempt. The Change Healthcare breach began with a stolen credential on a remote access portal that did not require multi-factor authentication, the kind of single exception a yes or no question hides. Control: ask for the list of exceptions, not a yes, and request evidence such as an identity provider policy export.
• Wrong respondent: sales or bid teams answer from a library. Control: require a named security owner to sign the critical answers.
• Scope sleight of hand: a valid SOC 2 report covers a different product or region. Control: match the system description to the service in your contract.
• Stale evidence: last year's report with no bridge letter. Control: track report period end dates and require bridge letters or updated reports.

Running the loop: ratings as triggers, assessments as context

1. Define trigger events. A score drop beyond an agreed threshold, a new critical exposed service, credentials in leak sources, a public breach disclosure or a change of ownership.
2. Route by tier. For critical and high suppliers, a trigger opens a targeted reassessment with three to five questions tied to the finding; for moderate, an email to the supplier contact; for low, a portfolio note.
3. Feed assessment results back. If the assessment shows a supplier hosts your data in a specific cloud service, add that asset to the watch list; if it shows heavy reliance on a subcontractor, watch that one too. See the Fourth-Party and Nth-Party Risk Guide.
4. Close or escalate. Record whether the finding was confirmed, disputed or remediated, with dates, so the next assessment starts from facts.

Worked scenario: three suppliers, one policy

A mid-sized insurer reviews three suppliers in the same week.

• Claims platform (critical): holds policyholder data and has an API integration into the core system. The insurer requires a SOC 2 Type 2 covering the claims service, a delta questionnaire, and evidence for four controls: phishing-resistant MFA for administrators, how integration tokens and keys are stored and rotated, backup restore tests and incident notification time. The rating is monitored daily. Three months later the rating flags a newly exposed remote desktop service. Because the supplier is critical, a call happens within two days; the supplier confirms a test server, removes it and supplies evidence. The finding is closed with dates.
• Marketing analytics SaaS (high): receives pseudonymised customer data through an OAuth connection. The insurer accepts an ISO/IEC 27001 certificate whose scope includes the service, asks five questions on token scope and data retention, and limits the OAuth grant. See the SaaS and OAuth App Governance Guide.
• Office catering (low): no data, no access. No questionnaire; the decision is recorded. The rating platform's low score for the caterer's website is ignored by design.

Where identity fits

Most of what ratings cannot see is identity: how the supplier authenticates its own administrators, how it secures service accounts and API keys that reach your systems, and how it stores OAuth tokens you granted. Put identity questions at the top of every critical and high assessment, require evidence for them, and inventory the non-human identities each supplier holds in your estate so you can revoke them quickly. The Third-Party Risk Management Guide covers the full lifecycle, including offboarding.

Implementation in phases

1. Inventory and tier. Build the supplier list from procurement, finance and identity provider app catalogues, and assign tiers using inherent risk.
2. Set evidence minimums per tier using the ladder, and publish them so business owners know what to expect.
3. Pilot ratings on critical and high suppliers first; measure false positives for a quarter before routing alerts automatically.
4. Rework the critical assessment around five to ten evidence-backed controls instead of hundreds of yes or no questions.
5. Wire triggers from ratings and breach news into reassessment workflow, with owners and service levels.
6. Review and roll back where needed. If triage load swamps the team, narrow ratings alerts to high-severity findings for top tiers only, and revisit thresholds quarterly.

Detection and measurement

Framework mapping

Practitioner checklist

• Tier every supplier on data, access and criticality before choosing any method.
• Publish the minimum evidence rung per tier and stop accepting self-attestation alone for critical suppliers.
• Confirm asset ownership before raising any rating finding with a supplier.
• Never let a good rating lower a tier or replace evidence.
• Ask for exceptions lists and configuration evidence for MFA, token storage and incident notification.
• Match every audit report's scope and period to the service you buy.
• Route rating drops and breach news into targeted reassessment for critical and high tiers.
• Record findings, disputes and remediation with dates for the next cycle.

Standards and references

• Federal Reserve, FDIC and OCC: Interagency Guidance on Third-Party Relationships: Risk Management
• NIST: SP 800-161 Rev. 1, Cybersecurity Supply Chain Risk Management Practices for Systems and Organizations
• US Chamber of Commerce: Principles for Fair and Accurate Security Ratings
• Cloud Security Alliance: STAR Levels
• NIST: Cybersecurity Framework (CSF) 2.0
• AICPA: System and Organization Controls (SOC) suite of services

Related NHI Mgmt Group resources: Third-Party Risk Management Guide · TPRM and Security Ratings Buyer's Guide · Security Questionnaires Guide · Supplier Security Audit Guide · Attack Surface Management Guide