Security leaders are often asked to choose between a bug bounty and a penetration test, when the two buy different things and fail in different ways. A penetration test buys documented assurance over an agreed scope from a contracted team. A bug bounty buys continuous attention from many unknown researchers on whatever is exposed and in scope, paid per valid finding. This guide compares them on coverage, cost, legal control, triage load, data exposure and compliance evidence, then gives a sequencing decision and worked scenarios. It is written for security managers, application security leads and the people who sign off testing budgets. For how to set up a disclosure programme itself, see the Bug Bounty and Vulnerability Disclosure Guide; for test methodology, see the Penetration Testing Guide.
Key takeaways
- They answer different questions. A penetration test asks "did our processes catch what a skilled team can find in this scope, on this date?"; a bounty asks "what can a crowd find on our exposed assets over time?"
- Sequence, do not substitute. Run a penetration test first, publish a vulnerability disclosure programme (VDP) next, and add a paid bounty only when triage and remediation capacity can absorb the volume.
- Cost behaves differently. A test is a fixed fee you can budget; a bounty is variable spend driven by how many valid findings exist, plus triage effort that is easy to underestimate.
- Control is the real trade-off. Contracted testers sign an agreement, work to rules of engagement and are named; bounty researchers are often anonymous and governed only by your published policy and safe harbour terms.
- Auditors usually want the test report. Frameworks such as CIS Control 18 and NIST SP 800-53 CA-8 describe penetration testing; a bounty supports, but rarely replaces, that evidence.
The short answer
If you need assurance evidence, coverage of internal systems or a defined methodology, commission a penetration test. If you already fix what tests find, have a working disclosure process and want continuous scrutiny of internet-facing applications, add a bug bounty. Every organisation with internet-facing systems should have a VDP regardless, because researchers will find issues whether or not you invite them, and a VDP gives them a lawful, low-friction route to tell you.
The NCSC frames a penetration test as "a method for gaining assurance in the security of an IT system" and says that, ideally, you already know what the testers will find, so new findings from experienced testers should be the exception. That framing is the key to the comparison: a test checks your vulnerability management, while a bounty is an additional discovery channel that only pays off when the basics already work.
The models compared
The table compares the four common options on the dimensions buyers actually argue about. The VDP is included because it is the bridge between the two.
| Dimension | Penetration test | VDP (unpaid) | Private bug bounty | Public bug bounty |
|---|---|---|---|---|
| Who tests | Named, contracted testers | Anyone who finds something | Invited researchers | Any researcher accepting the terms |
| Coverage model | Depth on an agreed scope, documented test cases | Opportunistic | Breadth and depth on a defined scope | Widest breadth on exposed scope |
| Timing | Point in time, scheduled window | Continuous | Continuous or time-boxed | Continuous |
| Payment | Fixed fee for effort (days) | None; thanks and credit | Per valid finding, by severity, plus platform or management fees if used | Same, at higher volume |
| Internal and authenticated targets | Yes, with credentials and access arranged | Rarely | Possible with provided test accounts | Usually limited to public-facing scope |
| Legal basis | Contract, statement of work, rules of engagement | Published policy with safe harbour | Programme terms plus safe harbour | Programme terms plus safe harbour |
| Output | Formal report with scope, method, findings and ratings | Individual reports | Individual reports, triaged | Individual reports, high volume |
| Compliance evidence | Strong: maps to testing requirements | Supports disclosure requirements | Supporting evidence | Supporting evidence |
| Main failure mode | Stale by the next release | Ignored reports | Narrow pool, quiet programme | Triage overload, duplicates, noise |
Coverage: depth versus breadth
A penetration test is scoped before it starts. The NCSC recommends that risk owners, technical staff and the test team scope it together and that the output is a plan covering technical boundaries, test types, timeframe and effort, and reporting. That gives depth: testers can be handed architecture diagrams, credentials for each role, source code for a clear box test, and access to internal networks. It also gives a record of what was tested, which matters as much as what was found. The limit is time. The NCSC notes a test "can only validate" that systems are not vulnerable to known issues on the day of the test, and test quality depends on the skill of the individual testers.
A bug bounty inverts this. Coverage is not planned, it emerges from where researchers choose to spend effort, and that follows reward size, ease of access and novelty. The result is strong breadth on popular, internet-facing web and mobile applications and weak coverage of anything that needs setup, specialist knowledge or authenticated access researchers cannot obtain. You cannot prove a given area was examined at all.
- Good bounty targets: customer-facing web applications, public APIs, mobile apps, marketing and acquisition domains that drift outside central control.
- Poor bounty targets: internal networks, Active Directory, OT and industrial systems, fragile legacy platforms, anything holding regulated data that a researcher could reach during testing.
- Penetration test only: internal and assumed-breach testing, cloud control plane reviews, Kubernetes, hardware and OT. See the Internal Network Penetration Testing Guide and the Cloud Penetration Testing Guide.
Cost and payment model
The two models move risk between buyer and tester in opposite directions. In a penetration test you pay for effort: if the testers find nothing, you still pay, and you have bought evidence of a clean result over a known scope. In a bounty you pay for results: if nothing valid is found, little is paid in rewards, but if your estate is full of easy issues, spend rises quickly and you pay for findings a basic scan would have shown.
| Cost element | Penetration test | Bug bounty |
|---|---|---|
| Direct spend | Day rate times days, agreed in advance | Rewards per valid finding, scaled by severity in a published reward table |
| Predictability | High: one purchase order per test | Low: depends on finding volume; needs a reward pool and a cap |
| Platform or management | Usually included in the engagement | Platform subscription or managed triage fee if a platform is used |
| Internal effort | Scoping, access set-up, retest coordination | Continuous triage, researcher communication, reward decisions, duplicates and disputes |
| Remediation | Batch of findings on one date | Steady stream; each report has a researcher waiting on a status update |
| Hidden cost | Retesting and scope creep | Paying for low-hanging issues, staff time on invalid reports, reputational cost of slow responses |
The practical rule: a bounty is cheapest per finding when the obvious issues are already fixed. Launching a bounty on an estate that has never been tested turns the reward pool into a very expensive vulnerability scan. For buying a test well, see the Penetration Testing Provider Buyer's Guide; for platform selection, see the Bug Bounty and VDP Platform Buyers Guide.
Legal control and safe harbour
A penetration test is authorised by contract. The statement of work and rules of engagement name the testers, the targets, the dates, the permitted techniques and the emergency contacts, and the provider carries professional liability. In the UK, the NCSC CHECK scheme provides NCSC-assured companies for central government, public sector bodies and critical national infrastructure. See the Red Team Rules of Engagement Template for the shape of these terms.
A bounty or VDP is authorised by a published policy that strangers accept by acting on it. That policy has to do the work a contract does: define scope, prohibited techniques (for example denial of service, social engineering, accessing or keeping other users' data) and what the organisation promises in return. CISA Binding Operational Directive 20-01 shows what a strong public commitment looks like: agencies commit not to recommend or pursue legal action against good-faith research that follows the policy and to treat it as authorised, and they must not limit testing to vetted parties. In the United States, the Department of Justice policy announced on 19 May 2022 directs that good-faith security research should not be charged under the Computer Fraud and Abuse Act. The open, vendor-neutral disclose.io project publishes safe harbour boilerplate, and notes that its material is not legal advice.
- What you cannot control in a bounty: who the researcher is, where they are, which tools they run, or how they store what they find. You can only set terms and enforce them after the fact.
- What a policy must state: scope, out-of-scope assets and techniques, how to report, what the finder can expect after reporting, disclosure timing and the safe harbour commitment. The NCSC toolkit adds: do not require the finder to sign an NDA.
- Data handling: require researchers to stop on first access to personal data, report it and delete it. BOD 20-01 requires agency policies to prohibit sharing discovered personal information with third parties.
Triage load and response obligations
The work a bounty creates is mostly internal. Every report must be reproduced, deduplicated, rated, routed to an owner, communicated back to the researcher and, for paid programmes, rewarded. Invalid and out-of-scope reports still need a timely answer. ISO/IEC 30111 covers this handling process and ISO/IEC 29147:2018 covers receiving reports and publishing remediation information; the ISO record shows 29147 was confirmed in 2024. See the ISO/IEC 29147 and 30111 Vulnerability Disclosure and Handling Guide.
Response expectations are increasingly written down. BOD 20-01 recommends resolving reports within about 90 days and tracks metrics such as median time to validate, remediate and first respond. In the UK, the Product Security and Telecommunications Infrastructure regime requires manufacturers of relevant connectable products to publish a point of contact and say when a reporter will receive an acknowledgement and status updates. A penetration test, by contrast, delivers its findings in one report on one date, and the triage load sits with the provider until hand-over.
Use one rating scheme across both channels so findings compete fairly for remediation time. See the CVSS v4 Guide and the Vulnerability Prioritisation Guide.
Data exposure to unknown researchers
A bounty deliberately invites people you have not vetted to probe production. That is the point, and also the risk. Researchers may reach real customer records, create accounts at scale, trigger fraud controls or leave test data behind. Contracted testers can be given dedicated test tenants and synthetic data; crowd researchers usually test in production unless you provide them with a safe environment.
- Provide test accounts and, where possible, a staging environment that mirrors production without real data.
- Tag researcher traffic by asking researchers to send an identifying header or use registered accounts, so the SOC can tell testing from attack.
- Keep regulated data out of scope until the programme has run privately and the data-handling terms are proven.
- Plan for secrets: exposed API keys, tokens and service account credentials are a common bounty finding. Agree in advance that researchers report without using them, and that the owner revokes and rotates rather than just deleting the file. See the Secrets Management Guide.
Compliance evidence
Auditors and customers generally ask for a penetration test report because frameworks describe testing in those terms. A bounty helps with disclosure requirements and shows continuous scrutiny, but it does not show that a defined scope was examined. Check the exact wording for your obligations in the Penetration Testing Requirements Map.
| Framework | Reference | Best satisfied by |
|---|---|---|
| NIST SP 800-53 Rev 5 | CA-8 Penetration Testing; CA-8(1) Independent Penetration Testing Agent or Team; CA-8(2) Red Team Exercises | Penetration test or red team |
| NIST SP 800-53 Rev 5 | RA-5(11) Public Disclosure Program | VDP or bug bounty |
| NIST SP 800-53 Rev 5 | SR-6(1) Testing and Analysis (supplier assessment) | Penetration test of supplier components |
| NIST CSF 2.0 | ID.RA-08 processes for receiving, analysing and responding to vulnerability disclosures | VDP and bounty triage process |
| NIST CSF 2.0 | ID.IM-02 improvements identified from security tests and exercises | Both, through lessons learned |
| CIS Controls v8.1 | 18.1 programme (IG2, IG3); 18.2 external test at least annually (IG2, IG3); 18.3 remediate; 18.4 validate measures (IG3); 18.5 internal test at least annually (IG3) | Penetration test |
| ISO/IEC 27001:2022 Annex A | 8.8 Management of technical vulnerabilities; 8.29 Security testing in development and acceptance | Both feed 8.8; testing supports 8.29 |
| CISA BOD 20-01 (US federal civilian) | Published VDP, scope growth, reporting metrics | VDP; a bounty may be linked as a separate policy |
CIS Safeguard 18.4 is worth noting: after each test, verify that controls detected the techniques used and adjust rules. A bounty rarely gives you that feedback loop, because researchers do not tell you how they got in beyond the steps to reproduce. See the CIS Control 18 Penetration Testing Guide.
Programme maturity prerequisites
Most failed bounties fail before launch, because the organisation was not ready. Use these gates.
| Gate | Penetration test | VDP | Paid bounty |
|---|---|---|---|
| Asset inventory for the scope | Needed for scoping | Needed to judge reports | Essential, including subdomains and acquisitions |
| Vulnerability scanning and patching in place | Recommended | Recommended | Essential, or rewards pay for scan findings |
| Named owner per asset | For remediation | Essential | Essential |
| Triage capacity | Provider-led | Low volume | Daily capacity, in-house or managed |
| Remediation service levels | Per report | Published target | Published target that is actually met |
| Legal review of terms | Contract | Policy and safe harbour | Policy, safe harbour, reward terms, tax and sanctions checks on payments |
| Previous test findings closed | Not needed for a first test | Helpful | Essential |
Measure readiness with the Attack Surface Management Guide for the inventory gate and the Vulnerability Management Guide for the remediation gate.
Decision guidance and sequencing
- Test first. Commission a penetration test of the most exposed and most valuable systems, close the findings and retest. This sets your baseline and gives assurance evidence.
- Publish a VDP. Add a policy, a reporting route and an RFC 9116 security.txt file. The NCSC toolkit recommends a secure web form and an Expires date less than a year ahead. Run it for long enough to measure your response times.
- Launch a private bounty. Invite a small group to a narrow scope with test accounts, a reward table and a monthly cap. Watch duplicates, validity rate and time to first response.
- Widen carefully. Grow scope and the researcher pool only when service levels hold. Move to public when triage keeps up with the private volume and rewards are competitive enough to attract effort.
- Keep testing. Continue annual and change-driven penetration tests, especially for internal, cloud and authenticated scope the bounty cannot reach. Use bounty findings to direct the next test's focus.
- Roll back if needed. Pause a bounty (return to private or VDP-only) if open valid reports age past your target, rather than letting researchers wait. Announce the change in the policy.
- Small organisations: an annual penetration test plus a VDP is usually enough. A paid bounty demands triage time a small team rarely has; a managed private programme with a small scope is the ceiling.
- Large organisations and product vendors: run all three. Product vendors face disclosure obligations (for example the UK PSTI regime) and benefit from crowd breadth across many releases.
- Regulated or public sector: lead with assured penetration testing (CHECK where applicable); add a VDP; consider a bounty only with tight data-handling terms.
Worked scenarios
Scenario 1: a mid-size SaaS provider facing customer security questionnaires. Customers ask for an annual third-party test report. The provider commissions an authenticated web application and API test of the production-equivalent tenant, fixes findings, and shares an attestation letter. It publishes a VDP with security.txt. A year later, with no aged reports, it launches a private bounty on the customer portal and public API with test tenants, keeping the admin console and payment flows in the penetration test scope only. The test report answers the questionnaires; the bounty catches issues introduced between tests.
Scenario 2: a retailer that skipped straight to a public bounty. Within weeks the team faces hundreds of reports about forgotten marketing subdomains, missing headers and an exposed staging server. Rewards for issues a scanner would have found drain the pool, and genuine critical reports wait behind noise. The fix is to step back to private, run attack surface discovery and a penetration test, retire or patch the forgotten assets, and relaunch with a tighter scope and clear out-of-scope list.
Detection and measurement
Log both channels into one vulnerability record so you can compare them on equal terms. Tag every finding with its source (test, VDP, bounty, internal), asset owner and rating. Feed researcher traffic identifiers into the SIEM so testing is visible but distinguishable.
- Valid finding rate: valid reports as a share of all reports, per channel. A falling bounty validity rate signals scope or reward problems.
- Time to first response and time to validate: the measures researchers judge you by; BOD 20-01 tracks both as medians.
- Time to remediate by severity: per channel, against your published target; count open valid reports older than 90 days.
- Cost per valid high or critical finding: test fee divided by findings versus rewards plus platform plus triage time.
- Repeat findings: issues the bounty finds that the last penetration test also reported, which point to remediation failure rather than testing failure.
- Detection rate for test techniques: share of penetration test actions the SOC detected, as CIS 18.4 asks.
Common failure patterns
- Using a bounty as a substitute for a test: no evidence of what was examined, no internal coverage, and auditors still ask for the report.
- Launching public before private: volume outruns triage and researchers lose patience, which leads to public disclosure on their timeline.
- Out-of-date scope: forgotten domains and acquired assets attract reports for systems no team owns, so nobody fixes them.
- Treating the test report as the end: the NCSC warns that mitigation should not be wholly outsourced to the test team; findings without owners stay open until the next test finds them again.
- Weak safe harbour wording: terms that demand NDAs, threaten legal action loosely or restrict disclosure indefinitely deter good-faith researchers while doing nothing to stop bad actors.
- No plan for exposed credentials: a reported key is deleted from the repository but never revoked, so it remains valid wherever it was copied.
Practitioner checklist
- Write down what each channel is for: assurance (test), reporting route (VDP), continuous discovery (bounty).
- Commission a scoped penetration test of critical systems and close its findings before paying rewards.
- Publish a VDP with safe harbour, scope and response expectations, plus security.txt on every domain.
- Confirm the asset inventory and owners for every in-scope system.
- Set a reward table, a monthly cap and a pause rule before launching any bounty.
- Start private with test accounts and a narrow scope; widen only when service levels hold.
- Keep internal, cloud control plane, OT and regulated-data systems in penetration test scope only.
- Use one rating scheme and one remediation queue for all findings.
- Report the six metrics above quarterly and compare cost per valid finding by channel.
- Review legal terms annually against current law and guidance.
Standards and references
- NCSC: Penetration testing
- NCSC: Vulnerability Disclosure Toolkit
- CISA: Binding Operational Directive 20-01, Develop and Publish a Vulnerability Disclosure Policy
- NIST: SP 800-115 Technical Guide to Information Security Testing and Assessment
- CIS: Control 18 Penetration Testing
- ISO: ISO/IEC 29147 Vulnerability disclosure
- US Department of Justice: Policy for charging cases under the Computer Fraud and Abuse Act
- disclose.io: Vulnerability disclosure and safe harbour resources
Related NHI Mgmt Group resources: Bug Bounty and Vulnerability Disclosure Guide, Penetration Testing Guide, Bug Bounty and VDP Platform Buyers Guide, Bug Bounty Programme Brief Template, Breach and Attack Simulation vs Penetration Testing.