Security Vendor Evaluation and RFP Guide: Running a Fair, Evidence-Based Selection

Most security tool selections are decided by the best demo and the most confident sales team, not by evidence that the product works in your environment. The result is shelfware, overlapping tools, and suppliers that hold sensitive access without anyone having checked how they protect it. This guide sets out a repeatable, vendor-neutral process for evaluating security products and services: defining requirements, writing an RFP that produces comparable answers, using standard questionnaires such as the SIG and CAIQ, designing a proof of concept that tests real risks, and scoring transparently. It draws on NIST SP 800-161r1 for cyber supply chain risk, UK public procurement guidance on award criteria, and CISA's Secure by Demand questions. It is written for security architects, CISOs, procurement and third-party risk teams who run or sign off a selection.

Key takeaways

• Requirements come before vendors. Write use cases, constraints and must-have criteria first, and fix the scoring model and weightings before any response is opened.
• Two questions, not one. Every security purchase needs a capability evaluation (does it work?) and a supplier risk assessment (is the supplier itself safe to depend on?), run in parallel with different owners.
• Standard questionnaires save time but prove little. The SIG and CAIQ give comparable self-attested answers; independent evidence such as audit reports and certifications, and your own testing, carry the weight.
• The proof of concept is the real evaluation. Test with your data, your integrations and scripted attack scenarios, with pass criteria agreed in advance, and measure operating effort, not just detection.
• Exit is part of the selection. Data return, credential revocation, contract termination and concentration risk belong in the RFP, because the cheapest time to negotiate them is before signature.

Plan the selection: scope, roles and timeline

A fair selection is mostly decided before the RFP is issued. NIST SP 800-161r1 advises that security considerations are identified early in the acquisition process, and that acquirers engage directly with suppliers to understand product characteristics and set expectations. In practice that means agreeing four things up front.

• Problem statement: one paragraph describing the risk or capability gap, the systems in scope, and what "better" looks like in measurable terms (for example, mean time to detect identity misuse below one hour, or 95 percent of internet-facing assets scanned weekly).
• Buy, extend or build: check what existing platforms already do. Many selections are avoidable because a licensed tool has an unused module. See the Security Tool Consolidation Guide.
• Evaluation panel: named people from security engineering, the operations team who will run the tool, architecture, procurement, legal, privacy and third-party risk. Each scorer declares conflicts of interest, including prior employment with or gifts from a bidder.
• Timeline and rules of engagement: a single point of contact for bidders, a clarification window with answers shared to all bidders, and a fixed response deadline. Side conversations between bidders and individual panel members undermine fairness and, in public procurement, can expose the buyer to challenge.

Small organisations can compress this into a two-week exercise with three or four people and a shortlist of two or three suppliers. Large or regulated organisations, and public bodies, need a documented process: in the UK the Procurement Act 2023 requires competitive awards to go to the "most advantageous tender" against published award criteria.

Write requirements that can be scored

Vague requirements ("must integrate with our SIEM") produce vague answers that every bidder can claim. Write each requirement so that a response can be verified and scored.

Classify each requirement as mandatory (pass/fail; failure eliminates the bid), scored (weighted) or informational (not scored). Keep mandatory items few and genuinely essential, otherwise you eliminate good bids on technicalities. The CISA and FBI Secure by Demand guide is a useful source of supplier-security requirements: it asks whether manufacturers provide standards-based SSO at no extra cost, enable MFA by default, have eliminated default passwords, include security logs in the baseline product, publish a vulnerability disclosure policy, and supply SBOMs in machine-readable formats.

Structure the RFP document

A well-structured RFP makes responses comparable line by line. A practical structure:

1. Instructions to bidders: timeline, contact point, format (a response spreadsheet with one row per requirement), page limits and confidentiality terms.
2. Background and problem statement: enough about your environment (scale, platforms, cloud providers, identity provider) to allow realistic answers, without disclosing sensitive architecture detail before an NDA.
3. Requirements matrix: numbered requirements with type (mandatory, scored, informational) and a fixed response scale such as "available now", "available with configuration", "roadmap with date", "third-party integration" or "not supported".
4. Supplier security and risk section: a standard questionnaire (SIG or CAIQ) plus your own targeted questions, and a request for evidence.
5. Commercials: a mandated pricing template covering licence metric, three to five years of cost, implementation, training, support tiers, overage charges and renewal caps.
6. Contract and exit terms: your standard terms, data processing agreement, security schedule and exit obligations, with bidders asked to list deviations.
7. Evaluation method: the criteria, weightings and scoring scale, published to bidders. UK guidance under the Procurement Act 2023 requires award criteria to be clear, measurable and proportionate, and their relative importance to be stated; publishing the method is good practice in private-sector selections too.

Ask bidders to separate generally available features from roadmap items, and treat roadmap answers as zero for scoring unless the contract will commit to them with a date and remedy.

Assess the supplier, not just the product

Security vendors frequently hold privileged access: agents with kernel or administrator rights, API keys to your cloud and SaaS platforms, or remote support channels into your estate. The BeyondTrust breach showed the risk: a compromised API key in the vendor's Remote Support SaaS infrastructure was used to reach US Treasury workstations. The supplier's own security is therefore a scored criterion, not a procurement formality.

Standard questionnaires

The CSA STAR Registry has two levels: Level 1 is a self-assessment submitted with the CAIQ; Level 2 adds third-party certification or attestation. Check the registry before sending a questionnaire, since a current entry can replace much of it. See the Security Questionnaires Guide and the Vendor Security Assessment Template.

Questions the standard sets do not ask

• Held credentials: which credentials (API keys, OAuth tokens, service account secrets) will the product store for our environment, where, how they are encrypted, who at the supplier can use them, and how fast can we revoke them all?
• Privileged components: what runs with kernel, root or domain administrator rights, how updates to those components are tested and staged, and whether we can control rollout rings.
• Support access: how supplier staff access our tenant or data, whether access needs our approval, and whether it is logged in a form we can see.
• Incident history: security incidents in the past three years affecting customers, and how and when customers were notified.
• Software supply chain: build integrity controls, signing of updates and SBOM availability.

Evidence over assertion

Weight independent evidence above self-attestation: an ISO/IEC 27001 certificate with a scope that covers the service you are buying, a SOC 2 Type II report for a recent period (read the exceptions and complementary user entity controls, not just the opinion), penetration test summaries, and a CSA STAR Level 2 entry. NIST SP 800-161r1 notes that reusing existing evidence such as a supplier's ISO 27001 certification reduces cost, though reassessment may still be needed. Scale effort to criticality: tier suppliers by access and data, and reserve deep assessment for those that could cause material harm. See the Third-Party Risk Management Guide.

Design a proof of concept that tests real risk

A proof of concept (PoC) is where claims meet your environment. Limit it to the two or three shortlisted bidders, give each the same scope, duration and test plan, and agree pass criteria in writing before it starts.

1. Define success criteria: for each scored requirement, state the measurement and threshold (for example, 90 percent of scripted test cases detected, alert-to-triage context available in under two minutes, under five hours a week of tuning effort).
2. Use a representative environment: real integrations, production-like data volumes and your standard builds. A vendor-hosted demo tenant tests the vendor's environment, not yours.
3. Script test cases: for detection products, emulate techniques relevant to your threat model, such as T1078 Valid Accounts, T1059.001 PowerShell and T1567 Exfiltration Over Web Service, using a controlled emulation framework, with expected outcomes recorded in advance.
4. Measure operating effort: have the operations team that will own the tool run it day to day for part of the PoC, recording false positives, tuning time and how often they needed vendor help.
5. Test failure and exit: what happens when the product's cloud service is unreachable (fail open or closed), how an update is rolled back, and whether a full data export works in the stated format.
6. Control the PoC itself: PoC deployments get real access. Use dedicated, least-privilege service accounts and API keys with an expiry date, record every credential issued, and revoke all of them when the PoC ends, whatever the outcome.
7. Record evidence: screenshots, exported logs and timings for each test case, so scores can be justified to stakeholders and, for public bodies, to unsuccessful bidders.

Score transparently and decide

Fix the weightings before responses arrive and do not change them after. A typical starting point for a security product, adjusted to the purchase:

• Scoring scale: use a defined 0 to 5 scale with written descriptors (0 = not met, 3 = met with configuration, 5 = met fully and demonstrated in PoC), so scorers interpret it the same way.
• Independent then moderated: each panel member scores alone first, then a moderation meeting agrees a consensus score with a written rationale. Large unexplained gaps between scorers usually reveal an unclear requirement.
• Price scoring: compare life-cycle cost, not first-year price. UK guidance under the Procurement Act 2023 states that tenders should generally be assessed on life-cycle cost and warns that scoring price relative to the lowest tender can harm value for money.
• Sensitivity check: before deciding, test whether the winner changes if weightings shift by five points. If it does, the decision is close and the panel should say so explicitly.
• Feedback: give unsuccessful bidders a summary of their scores. Public bodies in the UK must provide an assessment summary before publishing a contract award notice.

Contract, onboarding and exit

NIST SP 800-161r1 advises acquisition teams to include required controls in supplier agreements and, where risk warrants, to flow requirements down to sub-tier suppliers. For security products, the contract should cover at least:

• Security schedule: the controls the supplier committed to in its RFP response, made contractual, with a right to audit or to receive updated assurance reports annually.
• Incident notification: a defined time for the supplier to notify you of incidents affecting your data or service, and cooperation duties during your response.
• Change control for privileged components: notice of significant changes, staged update options, and the ability to defer updates for privileged agents.
• Exit: data return in a documented format within a set period, certified deletion, and your ability to revoke every credential and integration the product held. See the Third-Party Access Guide.
• Concentration: consider what happens if this supplier fails or ships a faulty update to every endpoint at once. See the Third-Party Concentration Risk Guide.

At onboarding, register the supplier in your third-party inventory with its criticality tier, the identities and credentials it holds, an internal owner, and a reassessment date.

Where identity fits

Many security tools are themselves large collections of non-human identities: cloud connectors with read or write roles, SaaS API tokens, directory service accounts and endpoint agents. Each is a path an attacker can use if the supplier, or the integration, is compromised. Score bidders on whether integrations support short-lived, scoped credentials (for example federated workload identity rather than long-lived access keys), whether permissions requested match the features you use, and whether the product exposes an inventory of the credentials it holds. Assign every integration identity an owner and an expiry. See the Third-Party Access Policy Template.

Measuring the selection process

• Requirements traceability: percentage of scored requirements with a verified PoC or evidence result (target: all mandatory and high-weight items).
• Post-deployment fit: share of PoC success criteria still met six months after go-live.
• Adoption: percentage of purchased capability (modules, licences) in active use after 12 months; low adoption signals shelfware.
• Supplier assurance currency: percentage of critical security suppliers with an assessment or assurance report less than 12 months old.
• Credential hygiene: number of PoC and integration credentials still active for suppliers not selected (target: zero).
• Cycle time: days from approved problem statement to contract, tracked so the process stays proportionate.

Framework mapping

For the full C-SCRM programme context, see the NIST SP 800-161 Cyber Supply Chain Risk Management Guide.

Common failure patterns

• Weightings set after the demos: criteria drift towards the favourite product. Fix and record weightings before opening responses.
• Roadmap scored as capability: features promised for "next quarter" win the evaluation and never ship. Score only generally available features unless the contract commits to dates and remedies.
• Questionnaire without evidence: a long self-attested questionnaire is filed and nobody reads the SOC 2 exceptions or checks the ISO certificate scope.
• Vendor-run PoC: the vendor configures the test, picks the attack scenarios and reports the results. Run the test plan yourself.
• PoC credentials left behind: API keys and admin accounts created for unsuccessful bidders remain active for months. Expire them by design and revoke at close.
• Ignoring the supplier's own exposure: a product that holds privileged keys into your estate is evaluated only on features, as illustrated by the BeyondTrust case above.

Practitioner checklist

• Write a measurable problem statement and confirm an existing tool cannot meet it.
• Name the evaluation panel, record conflicts of interest and set bidder communication rules.
• Write scorable requirements, classified as mandatory, scored or informational.
• Fix and publish criteria, weightings and the 0 to 5 scoring scale before responses arrive.
• Check the CSA STAR Registry and existing assurance reports before sending a SIG or CAIQ; add targeted questions on held credentials and privileged components.
• Run a PoC with the same scripted test plan for each finalist, in your environment, operated by your team.
• Issue PoC credentials with expiry dates and revoke all of them at the end.
• Score independently, moderate with written rationale, and run a weighting sensitivity check.
• Make RFP security commitments, incident notification and exit terms contractual.
• Register the supplier, its credentials and its reassessment date in the third-party inventory.

Standards and references

• NIST: SP 800-161 Rev. 1, Cybersecurity Supply Chain Risk Management Practices for Systems and Organizations
• NIST: Cybersecurity Framework 2.0
• NIST: SP 800-53 Rev. 5
• CISA: Secure by Demand Guide
• Shared Assessments: SIG Questionnaire
• Cloud Security Alliance: CAIQ v4 and STAR Level 1
• Cloud Security Alliance: Cloud Controls Matrix
• UK Government: Procurement Act 2023 Guidance, Assessing Competitive Tenders

Related NHI Mgmt Group resources: Third-Party Risk Management Guide · Security Questionnaires Guide · Security Tool Consolidation Guide · GRC Platform Buyer's Guide