Third-party risk tools are often bought to cut questionnaire workload, then judged on a score that nobody in the business can explain or verify. The market has two overlapping halves: TPRM platforms, which run the workflow of inventory, tiering, questionnaires, evidence review, findings and offboarding, and security ratings services, which score organisations from outside using internet-observable data. Many products now offer both, plus questionnaire exchanges where suppliers share completed assessments once with many customers. This vendor-neutral buyer's guide is for TPRM, GRC, procurement and security teams writing requirements or running an RFP. It explains what each approach can and cannot tell you, how to test ratings accuracy and dispute handling, which criteria to score, and how to run a proof of concept. Our Security Questionnaires Guide covers the questionnaire standards in depth.
Key takeaways
- Buy the workflow and the ratings for different jobs. A TPRM platform manages decisions and evidence across the supplier life cycle; a ratings service gives a cheap, continuous outside-in signal. Neither replaces the other.
- Outside-in data cannot see internal controls. A rating sees exposed services, certificates and email records; it cannot see MFA coverage, access reviews, backups or how a supplier stores your OAuth tokens.
- Test accuracy on suppliers you know. Attribution of IP addresses and domains to the wrong organisation is the most common ratings error; measure it during the trial and test the dispute process end to end.
- Hold providers to published fairness principles. The US Chamber of Commerce principles cover transparency, dispute and correction, validation, model change notice, independence from commercial relationships and confidentiality.
- The platform is itself a high-value third party. It stores your supplier contracts, assessment findings and integration credentials, so assess it as a tier 1 supplier.
Define requirements
Start from the programme, not the product. Agree tiers, assessment depth, risk acceptance authority and reporting before the first demo, using the life cycle stages in the US interagency guidance (planning, due diligence, contract negotiation, ongoing monitoring, termination) as a checklist of what the tool must support. Then write down:
- Population and volume: number of active suppliers, how many are tier 1 and 2, and assessments per year. Pricing often counts monitored suppliers or assessments, so volume drives cost.
- Regulatory outputs: for EU financial entities, the DORA register of information in the templates of Implementing Regulation (EU) 2024/2956; for US banks, the life cycle stages in the 2023 interagency guidance; for NIS2 entities, supply chain evidence under Article 21. See the DORA Guide and the NIS2 Directive Guide.
- Questionnaire standards: which you accept (SIG, CAIQ, HECVAT, VSA or your own) and whether you will accept answers shared through an exchange.
- Systems to integrate: procurement, contract management, accounts payable, the identity provider, ticketing, the GRC platform and the SIEM.
- Users: business owners raising requests, assessors, risk owners, suppliers responding through a portal, and auditors.
| Approach | What it does well | Limits | Best fit |
|---|---|---|---|
| TPRM workflow platform | Intake, tiering, questionnaires, document collection, findings, approvals, register and reporting | Only as current as the last assessment; supplier answers are self-reported | Any programme with more than a few dozen assessed suppliers or a regulatory register |
| Security ratings service | Continuous outside-in monitoring of many organisations at low marginal cost; trend and alerting | No view of internal controls; attribution errors; score methodology is proprietary | Monitoring tier 2 to 4 at scale; triggering reassessment of tier 1 |
| Questionnaire exchange | Suppliers complete once and share with many customers; faster cycle times | Answers may be stale or scoped to a different service; still self-attestation | Common SaaS and cloud suppliers already assessed by peers |
| Managed assessment service | Analysts chase suppliers and review evidence for you | Quality varies by analyst; you still own the decision | Teams with volume but few assessors |
Size changes the answer. A small organisation with 30 material suppliers may need only a spreadsheet, standard questionnaires and ratings for a short list. A regulated enterprise needs workflow, a register export, integrations and segregation of duties, with ratings used for monitoring rather than decisions.
Capability areas
What outside-in ratings can and cannot see
Ratings services scan and collect internet-observable data, attribute it to an organisation, and convert findings into a score or grade. Typical inputs are open ports and exposed services, TLS and certificate configuration, SPF, DKIM and DMARC records, DNS settings, software versions visible in banners, observations of infected hosts, and credentials or data found in leak sources. That helps spot neglected perimeter hygiene and sudden change, but says nothing about identity controls, segregation of customer data, secure development, backup recovery, logging or how the supplier protects tokens it holds for you. The Salesloft Drift breach, where stolen OAuth tokens of one integration opened many Salesforce tenants, is the kind of risk an external scan cannot detect.
Data accuracy, disputes and fairness
The commonest errors come from attribution: shared hosting, content delivery networks, cloud addresses that are reassigned, acquired companies and brand domains the supplier no longer runs. The US Chamber of Commerce Principles for Fair and Accurate Security Ratings, supported by ratings firms and organisations that rely on ratings, set a usable baseline:
- Transparency: "Any rated organization shall be allowed access to their individual rating", and providers should explain methodologies and data types.
- Dispute, Correction and Appeal: "Rated organizations shall have the right to challenge their rating and provide corrected or clarifying data", with disputed ratings marked as disputed until resolved.
- Accuracy and Validation: "Ratings should be empirical, data-driven, or notated as expert opinion", with methods validated against past performance.
- Model Governance: reasonable notice before methodology or data changes, with an explanation of the effect on existing ratings.
- Independence: "Commercial agreements, or the lack thereof, with rating companies shall not have direct impact on an organization's rating."
- Confidentiality: information shared in a dispute is protected, and individual ratings are not publicised.
Questionnaires, exchanges and evidence
Look for native support for standard questionnaires, conditional logic that skips irrelevant sections, evidence attachment per answer, and analyst review that records who accepted which answer. Exchanges and public registries reduce duplication: the Cloud Security Alliance STAR Registry, for example, "is a publicly accessible registry that documents the security and privacy controls" of cloud services, with Level 1 self-assessment and Level 2 third-party audit or certification. Check the date and service scope of any shared answer before relying on it.
Workflow, integrations and the platform's own security
Workflow should cover intake from procurement, inherent risk tiering, assessment, findings with owners and due dates, risk acceptance with expiry, reassessment scheduling and offboarding tasks. Integrations with the identity provider let you link supplier accounts and OAuth apps to supplier records. Those integrations and the platform's API keys are non-human identities with access to sensitive data: the OWASP NHI3:2025 Vulnerable Third-Party NHI guidance is to "Grant only the minimum permissions required (principle of least privilege) and regularly audit or revoke unused access." See the SaaS and OAuth App Governance Guide.
Evaluation criteria
Weight criteria to your requirements. An illustrative split for a workflow-led purchase is 30% workflow and register, 20% integrations, 20% ratings accuracy and monitoring, 15% platform security and data, 15% supplier experience and reporting.
| Criterion | What good looks like | Weak signs |
|---|---|---|
| Inventory and tiering | Configurable inherent risk questions; tier recalculated on scope change; import from procurement and SSO or OAuth discovery | Tier fixed at onboarding; supplier list maintained by hand |
| Assessment workflow | Standard questionnaires, conditional logic, evidence per answer, findings with owners and due dates, risk acceptance with expiry | Questionnaires emailed as spreadsheets; no findings tracking |
| Regulatory register | Contract, function and subcontractor data that can produce the DORA register of information or an equivalent outsourcing register | Register built outside the tool each year |
| Ratings methodology | Published factor list, data types and weighting; customer-visible findings with evidence and timestamps | Score with no drill-down to the observation |
| Accuracy and disputes | Rated organisations can view and dispute free of charge; disputed findings flagged; dispute turnaround measured | Paid access needed to correct data; disputes silently closed |
| Fourth-party visibility | Subcontractor capture from suppliers; concentration reporting across suppliers | Fourth parties inferred only from DNS or web technologies |
| Integrations | Procurement, contract, IdP, ticketing, GRC and SIEM connectors; full API | CSV import only; no outbound alerts |
| Platform security | SSO, enforced MFA, SCIM, role-based access, tenant audit log, current independent assurance, published vulnerability disclosure policy | Shared supplier portal logins; no customer audit log |
| Data and exit | Named hosting regions, retention controls, full export of suppliers, assessments, evidence and history | Assessment content owned by the vendor after exit |
Questions to ask vendors
- Show the evidence behind three findings on a supplier we name: the observation, the date, and how the asset was attributed to that organisation.
- How can a rated organisation see and dispute its rating, at what cost, and what is your median dispute resolution time?
- Which factors drive the score, how are they weighted, and how much notice do customers get before a methodology change?
- Does any commercial relationship with a rated organisation affect its score or its access to remediation help?
- How fresh are shared questionnaire answers in your exchange, and how do you show which service and date they cover?
- Can you produce the DORA register of information, or our regulator's outsourcing register, from data held in the platform?
- Which permissions do your identity provider and SaaS connectors request, are they read-only, and do they use short-lived credentials?
- Do you offer SSO without extra cost, MFA on by default and security logs retained for customers, as CISA's Secure by Demand Guide asks of software suppliers?
- What independent assurance covers your platform, and what security incidents have you disclosed?
- At contract end, what data do we receive, in which formats, and when is it deleted?
Red flags
- A single score presented as a measure of a supplier's overall security, or as grounds to approve a tier 1 supplier without further assessment.
- No free route for rated organisations to see and correct their data.
- Methodology changes that move scores across your whole portfolio without notice.
- Findings you cannot trace to a dated observation and an attributed asset.
- Supplier portals with shared logins, no MFA, or document uploads without malware scanning.
- Connectors that require administrator rights or long-lived API keys.
- Export limited to reports, so your assessment history stays behind at exit.
Proof of concept
- Agree pass criteria first: weights, accuracy thresholds and the workflows to test, signed off before the first demo.
- Test ratings on known suppliers: pick 20 suppliers, including your own organisation and suppliers whose environments you know well. Have each finding checked by someone who can confirm whether the asset belongs to them, and record the false attribution rate.
- Run a dispute: raise a real correction for your own organisation and time it to resolution.
- Run the workflow end to end: intake a new SaaS supplier, tier it, send a standard questionnaire, raise a finding, accept a risk with an expiry, and offboard a test supplier including access revocation tasks.
- Connect with least privilege: create dedicated, read-only integration identities with named owners and record every permission requested.
- Produce a register: generate your regulatory register or a board report from trial data and check completeness.
- Test exit and roll back: export all data, revoke the integration credentials, confirm connectors stop, and request deletion confirmation.
Measure the trial with figures you can keep tracking: false attribution rate, dispute turnaround, assessment cycle time by tier, percentage of findings with owners and due dates, and percentage of suppliers with a current tier.
How NHI Mgmt Group can help
We provide independent requirements, RFP and evaluation support. Browse vendors in our products directory or contact us.
Related NHI Mgmt Group resources: Security Questionnaires Guide · Third-Party Access Guide · GRC Platform Buyer's Guide · Attack Surface Management Guide