Penetration Testing Methodologies Compared: Which to Require for Each Test

Most penetration test proposals name a methodology, but buyers rarely check whether it fits the test they are paying for or what evidence it will produce. PTES, OSSTMM 3, the OWASP Web Security Testing Guide (WSTG) and NIST SP 800-115 were written for different purposes: an engagement lifecycle, a measurement discipline, a web test catalogue and a federal testing reference. CREST guidance sits alongside them as a delivery and assurance standard rather than a methodology. This guide compares them for the people who scope, buy and accept tests: security assurance leads, procurement, auditors and CISOs. It covers what each one is for, the evidence each produces, how to combine them by test type, and what to write into a statement of work. It does not explain how to run tests.

Key takeaways

• They answer different questions. PTES structures the engagement, OSSTMM measures operational security, the WSTG lists what to test in a web application, and SP 800-115 describes the phases and techniques a testing programme uses.
• Combine, do not choose one. A typical web test uses an engagement lifecycle (PTES or SP 800-115) plus the WSTG for coverage; an infrastructure test uses SP 800-115 or PTES with ATT&CK tactics for coverage.
• Buy the evidence, not the name. Ask what each methodology makes the provider hand over: test case IDs, coverage achieved, a statement of what was not tested, logs and a reproducible finding.
• Check the dates. SP 800-115 is from 2008, OSSTMM 3.02 from 2010 and the PTES has not been maintained since 2011 according to CREST, while the WSTG's latest release is v4.2 with 5.0 in development. Age is not disqualifying, but coverage of cloud and identity has to come from elsewhere.
• Assurance comes from the delivery standard. CREST's Defensible Penetration Test states that it does not define a methodology; it sets minimum scoping, delivery, sign-off and report contents, which is what a buyer can actually audit.

The short answer

Methodologies are listed by role, not ranked. Pick by the question you need answered.

For the general programme view (test types, scoping and providers) see the Penetration Testing Guide.

NIST SP 800-115: the federal reference

What problem it addresses: SP 800-115 helps organisations plan and run technical security tests and examinations, analyse findings and develop mitigation. NIST is explicit that it is not a comprehensive testing programme, only an overview of key elements.

Structure in words: Section 2.2 groups techniques into three categories: review techniques (manual examination of documents, configurations and logs), target identification and analysis (finding hosts, ports and services), and target vulnerability validation (confirming vulnerabilities, including penetration testing and social engineering). Section 2.4 adds viewpoints: external or internal, and overt (with IT staff knowledge) or covert (management approval only). Section 5.2.1 gives the four-phase penetration test model in Figure 5-1: planning (rules agreed, approval documented, no testing), discovery (information gathering and vulnerability analysis), attack (verifying vulnerabilities, looping back to discovery as access grows) and reporting, which runs alongside the other three. Appendix B is a rules of engagement template.

Where it is required: FedRAMP Penetration Test Guidance v3.0 (June 2022) names SP 800-115 as the required reference, requires the test plan and rules of engagement to follow Appendix B with authorising official approval, requires manual testing rather than scanning alone, and sets six attack vectors, from external to corporate through tenant to tenant. Detail is in the NIST SP 800-115 Security Testing and Assessment Guide and the FedRAMP Guide.

Failure modes: treating the 2008 technique list as current coverage for cloud control planes, identity providers or containers; citing it without the Appendix B rules of engagement; and accepting "SP 800-115 based" with no statement of which technique categories were in scope.

Choose it when: a regulator or customer names it, you need a public reference that auditors recognise, or you are designing an internal testing programme and want the review, identification and validation split to organise it.

PTES: the engagement lifecycle

What problem it addresses: giving buyers and testers a common language for how an engagement runs from first contact to report. Its seven sections are pre-engagement interactions, intelligence gathering, threat modelling, vulnerability analysis, exploitation, post exploitation and reporting. The standard says it does not provide technical guidelines; a separate PTES Technical Guidelines document filled that role.

Structure in words: the value for a buyer is at the ends. Pre-engagement forces agreement on scope, goals, timing, communication and permissions before testing. Threat modelling ties testing to the assets and adversaries that matter to the business. Reporting separates the executive summary from the technical findings. The middle sections describe the testing flow without prescribing test cases.

Failure modes: CREST's Defensible Penetration Test notes PTES has not been maintained since 2011, so it says nothing specific about cloud, SaaS or modern identity attack paths; providers can claim PTES alignment while skipping threat modelling entirely; and because it has no test catalogue, coverage cannot be checked against it.

Choose it when: you want a clear engagement structure for commercial tests, particularly where a threat-modelling step is needed to justify scope. Pair it with a coverage source: the WSTG for web, ATT&CK tactics for infrastructure.

OSSTMM 3: the measurement discipline

What problem it addresses: producing factual, repeatable measurements of operational security rather than a list of vulnerabilities. Version 3.02 (ISECOM, December 2010) deliberately drops risk as a metric in favour of measuring the attack surface.

Structure in words: OSSTMM tests five channels: human and physical (physical security), wireless (spectrum security), and telecommunications and data networks (communications security). Porosity counts interactive points as visibility, access or trust. Controls are ten, in two classes: interactive (authentication, indemnification, resilience, subjugation, continuity) and process (non-repudiation, confidentiality, privacy, integrity, alarm). Limitations are classed as vulnerability, weakness, concern, exposure and anomaly, which the manual says are not a severity hierarchy. These combine into the rav, a scale measurement of attack surface where 100 is a perfect balance of controls against porosity and limitations. Section 2.3 defines test types by what each side knows, from blind and double blind through gray box, double gray box and tandem to reversal, which it notes is often called a red team exercise. Results go in a STAR signed by the analysts, which can be submitted to ISECOM for certification.

Failure modes: buying an "OSSTMM test" without a STAR or rav values, which means the measurement was not done; expecting a vulnerability list in the familiar severity format; and the learning cost, since the vocabulary is unfamiliar to most report readers.

Choose it when: the scope spans people, premises, wireless or telecoms as well as networks; you want to compare the same environment year on year with one metric; or you need the test type definitions in a contract to state exactly what the target team will know.

OWASP WSTG: the web coverage catalogue

What problem it addresses: defining what a web application test should cover. The WSTG is a guide to testing web applications and web services, organised as test scenarios with IDs in the form WSTG-<category>-<number>, such as WSTG-INFO-02. The latest release is v4.2 (December 2020) and 5.0 is in development. Because IDs can change between versions, OWASP recommends the versioned form, for example WSTG-v42-INFO-02, and versioned links.

Structure in words: it is a catalogue, not a lifecycle. It tells the tester what to look at in information gathering, configuration, identity and session management, authorisation, input validation, error handling, cryptography, business logic and client-side code. It does not tell the buyer how to scope, staff or sign off the engagement.

Failure modes: unversioned references, so nobody can check coverage later; using the WSTG for native mobile apps or non-web APIs where other OWASP material fits better; and reports that cite the WSTG but give no coverage table, so untested categories are invisible. Detail is in the OWASP WSTG Guide; the matching requirement standard is in the OWASP ASVS Guide and mobile in OWASP MASVS.

Choose it when: anything is delivered over HTTP. Require findings and a coverage table mapped to versioned IDs.

CREST guidance: the delivery and assurance layer

What problem it addresses: giving buyers something auditable regardless of which methodology a provider uses. CREST's Defensible Penetration Test (2023) is a minimum specification and says it does not define a methodology or a list of items to test; delivery follows the provider's own CREST-accredited methodology.

Structure in words: three gates. Scoping must set out the full attack surface relevant to the target and is agreed between buyer and provider. Delivery must cover everything scoped and record constraints. Sign-off is a formal attestation that the accredited methodology was followed and the agreed scope addressed. The minimum report contents include goals, scope with exclusions and coverage achieved, full results, a timeline with logs available on request, enough detail to reproduce each issue, CVSS severity, remediation advice, a statement of totality against scope and evidence that staff were qualified, with SFIA level 4 for scoping and delivery and level 5 for sign-off. CREST's A Guide to Penetration Testing (2022) adds the programme view: 22 steps across preparation, testing and follow-up, naming OSSTMM, OWASP, NIST SP 800-115 and PTES as recognised approaches and noting that suppliers often use their own methodology while showing compliance with standard ones. See CREST Certifications.

Failure modes: assuming accreditation means the scope was right (scope remains the buyer's job); and accepting a report without the statement of totality, which is the line that tells you what was not tested.

Choose it when: you need defensible evidence for a board, auditor or customer that a test was competent and complete against scope.

Which methodology for which test

NCSC guidance does not prescribe a methodology, and none of the four covers every test type. Combine an engagement lifecycle with a coverage source.

Evidence outputs to require

The methodology name matters less than the evidence it obliges the provider to deliver. NCSC adds one item most buyers miss: the tester's opinion on how accurate your own vulnerability assessment was.

How to read the result is covered in How to Read a Penetration Test Report.

Worked scenarios

Scenario 1: a SaaS provider with a customer-facing web app and a cloud tenant

The customer contract asks for "an annual penetration test to an industry-recognised methodology". The provider's statement of work specifies SP 800-115 phases and Appendix B rules of engagement for the whole engagement, the WSTG v4.2 for the application with a coverage table by category, and for the tenant a chosen set of ATT&CK tactics with the reasons for exclusions. The report must meet the CREST Defensible Penetration Test minimum contents, including the statement of totality.

Scenario 2: a retailer with stores, distribution centres and a head office

The board wants to know whether security is improving, not another list of findings. The retailer commissions an OSSTMM test of the physical, wireless and data network channels at three representative sites, specifying the double gray box test type so staff know the timeframe but not the vectors. The STAR and rav values per channel become the baseline. Separately, the e-commerce site gets a WSTG-based web test. Next year the same scope is repeated for comparison.

Decision guidance for buyers

• Regulated or federal: start from what the rule names. FedRAMP requires SP 800-115; PCI DSS requirement 11.4.1 expects a documented methodology based on industry-accepted approaches. See the Penetration Testing Requirements Map.
• Small organisations: one lifecycle (SP 800-115 is free and recognised) plus the WSTG for web is enough. Spend effort on scope and on fixing findings, not on methodology debate.
• Large organisations with several providers: standardise the evidence, not the provider's internal method: one report template with WSTG mapping, coverage, statement of totality and a common severity scale, so results compare across a testing panel.
• Proprietary methodologies: acceptable if mapped. CREST notes suppliers often use their own while showing compliance with standard ones; ask for the mapping document.
• Identity in scope: the WSTG covers application identity and session management, but none of the four treats identity providers, service accounts or API keys as estate-wide test targets. Name them in scope explicitly and control the test credentials you issue. See the Service Account Security Guide.

How to put it into a statement of work

1. State the question: compliance evidence, assurance of a new system, or trend measurement. This decides lifecycle and coverage source.
2. Name the lifecycle and coverage source by version: for example SP 800-115 Section 5.2.1 phases and WSTG v4.2, never "industry best practice".
3. Define the test type in words both sides understand: what the testers know and what your team knows, using SP 800-115 overt or covert or the OSSTMM test types.
4. Attach the rules of engagement: Appendix B as a base, with emergency stop, critical finding notification and third-party permissions.
5. List the evidence outputs: from the evidence table, as acceptance criteria for payment.
6. Agree retest terms: CREST's programme guide expects retesting after mitigation to be in scope statements, and testing of critical systems at least yearly and after major change.
7. Review on acceptance: check coverage and the statement of totality before signing off, and feed findings to your vulnerability management process. Use the Penetration Testing Services RFP Template to carry these into procurement.

Detection and measurement

• Coverage rate: share of scoped test categories (WSTG categories, ATT&CK tactics or OSSTMM channels) actually tested, from the coverage table.
• Exclusion trend: number of items in the statement of totality marked not tested, per engagement.
• Detection of tester activity: share of tester actions in the timeline that your monitoring detected and alerted on.
• Previously unknown findings: share of findings your own scanning had not already found, the measure NCSC asks testers to comment on.
• Time to remediate and retest pass rate: by severity, tracked in the Penetration Test Remediation Tracker Template.

Framework mapping

Common failure patterns

• A name with no version: "OWASP methodology" or "NIST based" with no document or version, so coverage cannot be checked.
• Lifecycle without coverage: a PTES-structured report with no test catalogue behind it, leaving gaps invisible.
• No statement of what was not tested: constraints and exclusions are discussed during the test and never written down.
• Measurement methods bought, list delivered: an OSSTMM engagement that produces no STAR or rav values.
• Dated methods trusted for modern estates: 2008 and 2010 documents used as the only coverage source for cloud and identity providers.
• Testing treated as vulnerability management: NCSC warns penetration testing should not be the primary way of finding vulnerabilities.

Practitioner checklist

• Write down the question the test must answer before choosing a methodology.
• Name a lifecycle and a coverage source, each with its version, in the statement of work.
• Require versioned WSTG IDs for every web finding and a coverage table.
• Attach rules of engagement based on SP 800-115 Appendix B, with critical finding notification.
• Require the CREST Defensible Penetration Test minimum report contents, including a statement of totality.
• Ask for the mapping of any proprietary methodology to a public one.
• Name identity providers, service accounts and API keys explicitly in scope.
• Repeat scope and test type for OSSTMM measurement so values compare year on year.

Standards and references

• NIST: SP 800-115, Technical Guide to Information Security Testing and Assessment
• ISECOM: Open Source Security Testing Methodology Manual (OSSTMM 3)
• OWASP: Web Security Testing Guide
• The PTES Team: Penetration Testing Execution Standard (Read the Docs edition)
• CREST: Defensible Penetration Test
• CREST: A Guide to Penetration Testing
• FedRAMP: Penetration Test Guidance
• NCSC: Penetration testing

Related NHI Mgmt Group resources: Penetration Testing Guide · NIST SP 800-115 Security Testing and Assessment Guide · OWASP WSTG Guide · Penetration Test Scoping Checklist · Penetration Testing Provider Buyer's Guide