Buyer’s Guide Methodology and Scoring Template: A Repeatable Way to Score Security Products

Security product selections are often decided by the best demonstration or the loudest stakeholder, and six months later nobody can explain why the chosen product won. A written methodology with fixed weights, a defined scoring scale and an evidence log makes the decision repeatable, defensible to auditors and fair to suppliers. This template gives [Organisation] a ready-to-use procedure for evaluating any security product or managed service, from an identity platform to an MFT server or an MSSP. It covers scope, roles, must-have gates, criteria and weights, the scoring scale, evidence, proof of concept scoring, risk and commercial review, and the decision record. It is for security architects, procurement leads and evaluation panels. For the wider RFP process, see the Security Vendor Evaluation and RFP Guide.

How to use this template

• Copy and adapt: replace every [placeholder] with your own values, delete criteria that do not apply, and keep the section numbering so evaluation records can reference it.
• Fix the method before you meet suppliers: sections 3 to 5 (gates, criteria, weights and scale) must be approved by the evaluation owner before any supplier presentation or quote is seen. Changing weights after seeing scores is the most common way evaluations lose credibility.
• Score evidence, not claims: each score in the scoring sheet must cite an evidence reference (document, demonstration recording, proof of concept result or reference call note).
• Scale the effort to the risk: a low-risk tool bought for one team may need only sections 3, 4, 5 and 9; a platform that will hold privileged credentials or customer data needs every section.
• Use it with the buyer's guides: the capability tables in the NHI Mgmt Group buyer's guides, such as the Managed Security Service Provider Buyers Guide or the Secure File Transfer Buyers Guide, can be pasted into section 4 as starting criteria.

1. Purpose and scope

1.1 This methodology defines how [Organisation] evaluates and selects security products and services so that decisions are consistent, evidence-based and auditable.

1.2 It applies to any purchase, renewal or replacement of a security product or managed security service with a total contract value above [threshold, for example GBP 25,000] or that will [hold privileged access, process personal data, or be internet-facing].

1.3 Each evaluation records: the business problem, the systems and data in scope, the expected contract term, the budget range and the target decision date of [date].

1.4 Out of scope: [emergency purchases under the incident response procedure, which are reviewed retrospectively using sections 6 and 9 within 90 days].

2. Roles and governance

2.1 Each panel member signs a conflict of interest declaration before scoring. Any relationship with a supplier (previous employment, shares, gifts above [value]) is recorded and the evaluation owner decides whether the member may score that supplier.

2.2 All supplier contact goes through the procurement lead. Questions and answers are shared with all shortlisted suppliers where they clarify the requirement.

2.3 Panel members score independently first, then meet to moderate. The moderated score and the reason for any change are recorded.

3. Requirements and must-have gates

3.1 Requirements are split into must-have gates and scored criteria. A supplier that fails any gate is excluded, whatever its score.

3.2 Gates are pass or fail, testable and few (normally five to ten). Example gates:

• Identity: supports single sign-on with [identity provider] via SAML 2.0 or OpenID Connect, and phishing-resistant MFA for administrators.
• Data location: customer data stored and processed in [regions].
• Logging: exports administrative and security events to [SIEM] in near real time.
• Assurance: holds [ISO/IEC 27001 certification covering the service, or a current SOC 2 Type II report].
• Vulnerability handling: publishes security advisories and has a coordinated vulnerability disclosure policy.
• Exit: provides export of customer data and configuration in a documented format at contract end.

3.3 Each scored requirement is written so two evaluators would reach the same score from the same evidence. Avoid "intuitive", "robust" and "best of breed"; prefer "creates a ticket in [ITSM] with asset owner populated".

4. Criteria and weights

4.1 Criteria are grouped into categories with weights that total 100%. Weights are approved by the evaluation owner before supplier engagement and recorded with the approval date.

4.2 Within each category, list three to eight criteria and give each a share of the category weight. Criteria that duplicate a gate are not scored again.

4.3 Where the product will hold credentials, tokens or privileged access, add an identity criterion: the supplier lists every service account, API key, OAuth application and agent it will create in [Organisation]'s estate, with permissions and rotation. See the Third-Party Access Guide.

5. Scoring scale

5.1 Every criterion is scored on the scale below. Half points are not used.

5.2 Weighted score for a criterion = score / 5 x criterion weight. The supplier's total is the sum of weighted scores, out of 100.

5.3 Worked example: a criterion weighted 6% scored 4 contributes 4 / 5 x 6 = 4.8 points.

5.4 Roadmap features score no higher than 2 unless the supplier commits to a delivery date in the contract.

5.5 Where totals of the top two suppliers are within [5] points, the evaluation owner treats them as equivalent and decides on risk, references and commercial terms, recording the reason.

6. Evidence and proof of concept

6.1 Every score carries an evidence reference in the scoring sheet: [EV-001 document, EV-002 demo recording, EV-003 POC result, EV-004 reference call].

6.2 Shortlisted suppliers (normally two or three) take part in a proof of concept of [two to six] weeks in a representative environment with test data or approved production data.

6.3 Test cases are written before the proof of concept starts, each with an expected result and pass criteria. Example test cases:

1. [Deploy to the pilot group within the agreed time without vendor engineering changes.]
2. [Detect or block the agreed benign test scenario and alert within X minutes.]
3. [Export logs to the SIEM in a parsed format.]
4. [Grant and revoke administrative access through SSO and confirm in logs.]
5. [Restore service or configuration from backup.]
6. [Remove the product and confirm no agents, accounts or credentials remain.]

6.4 Results are recorded as measured values (minutes, percentages, counts), not impressions. Suppliers may observe but do not operate the product during scored tests unless the test is about vendor support.

6.5 Reference calls use the same question list for each supplier and include at least one customer of similar size and sector.

7. Security and supplier risk review

7.1 The security risk reviewer assesses each shortlisted supplier using [Organisation]'s third-party risk process and records a risk rating of [low, medium, high].

7.2 The review covers: assurance reports and their scope, vulnerability disclosure history, data handling and location, sub-processors, the supplier's own access to [Organisation]'s systems, incident notification commitments and business continuity. See the Third-Party Risk Management Guide and the How to Read a SOC 2 Report Guide.

7.3 A high residual risk rating requires either mitigation agreed in the contract or a documented risk acceptance by [risk owner] before award.

8. Commercial review

8.1 Costs are compared as total cost of ownership over [3 or 5] years, including licences, infrastructure, implementation, training, internal staff time, support tiers, expected growth and exit costs.

8.2 Pricing units (per user, endpoint, gigabyte, asset or event) are normalised to [Organisation]'s expected volumes for year one and year three.

8.3 Contract terms checked before award: price caps on renewal, termination rights, service levels and credits, data return and deletion, audit rights, security incident notification within [hours], and liability. See the Third-Party Security Contract Clauses Template.

9. Decision record

9.1 The evaluation owner signs a decision record containing:

• the approved method, weights and gates, with approval date;
• suppliers invited, excluded at gates (with reason) and shortlisted;
• individual and moderated scores, with evidence references;
• proof of concept results and reference call notes;
• security risk rating and any risk acceptance;
• total cost of ownership comparison;
• the recommendation, the reasons, and any conditions for award;
• conflict of interest declarations.

9.2 The decision record is retained for [contract term plus 6 years] and made available to internal audit.

9.3 Unsuccessful suppliers receive a short debrief on request, based on the recorded scores.

9.4 At [12 months] after go-live, the evaluation owner compares actual outcomes against the proof of concept results and records lessons for the next evaluation.

Implementation guidance

Start with one evaluation. Pilot the template on the next renewal, refine the criteria library, then make it mandatory above the threshold in section 1.2.

Keep a criteria library. Store reusable criteria by product category so each evaluation starts from proven wording rather than a blank page.

Watch for common failure patterns:

• Weights set after scoring: the method is changed to favour a preferred supplier. Record the weight approval date and lock the sheet.
• Feature counting: long criteria lists where trivial features carry the same weight as the core use case. Cap criteria per category.
• Demo-driven scores: scripted demonstrations scored as evidence. Only score demonstrations run against your own scenarios.
• Ignoring the exit: no test of data export or uninstall, so lock-in is discovered at renewal.
• Supplier access left behind: pilot accounts, API keys and agents remain after the proof of concept. Section 6.3 test 6 exists for this reason.

Measure the process: track evaluations completed within the planned time, percentage of scores with an evidence reference, number of weight changes after approval (target zero), and the gap between proof of concept results and measured performance after 12 months.

Framework alignment: this procedure supports NIST CSF 2.0 GV.SC-04, GV.SC-05 and GV.SC-06 (knowing and prioritising suppliers, putting security requirements into contracts, and due diligence before entering a relationship), and NIST SP 800-53 Rev. 5 SA-4 Acquisition Process, SR-5 Acquisition Strategies, Tools, and Methods and SR-6 Supplier Assessments and Reviews. See NIST: Cybersecurity Framework 2.0 and NIST: SP 800-53 Rev. 5. For UK organisations, the NCSC supply chain security guidance sets out principles for understanding supplier risk and establishing control before contracting.

Related NHI Mgmt Group resources: Security Vendor Evaluation and RFP Guide · Third-Party Risk Management Guide · Third-Party Security Contract Clauses Template · Managed Security Service Provider Buyers Guide