Test Discovery

How test discovery works

Test discovery is the part of a build or test runner that decides which files count as tests and should be executed. It usually follows naming patterns, directory rules, or explicit configuration, so the exact discovery logic is often more important than the test code itself when you are trying to understand what will actually run.

Because discovery happens before execution, it defines the boundary between intended test assets and everything else in the repository. A small change to that boundary can alter coverage, introduce noise, or cause files to run that the team never meant to treat as tests.

Why recursive discovery changes the trust boundary

Recursive discovery is convenient because it reduces configuration and picks up new tests automatically, but it also expands the search surface. If the tool walks the whole tree, then hidden folders, vendor content, generated output, or copied examples may be treated as runnable tests when they should not be.

That matters because discovery is not just file enumeration, it is a trust decision. Once an untrusted path is included, the build process may execute code from a location that was never reviewed as first-class project source, which makes repository layout part of the security model rather than a purely organisational choice.

Common failure modes in build and test tooling

The main failure modes are overbroad path matching, weak exclusion rules, and assumptions that all recursively found files are safe to execute. Hidden directories are especially risky because many tools skip them by default in one context but include them in another, and a team may not notice the difference until an unexpected file runs.

Another common issue is test naming collisions. A helper script, fixture, or sample file can match a test pattern by accident, especially in large monorepos or mixed-language repositories, so the test runner ends up evaluating code that was never designed for that lifecycle. The same problem can appear when nested project structures inherit discovery settings from a parent directory.

What discovery means for secure software delivery

Test discovery is a software assurance concern because it affects the integrity of validation itself. If the wrong files are selected, a passing test run can give false confidence, while a malicious or simply misplaced file can influence the build in ways that are hard to spot during review.

For teams that depend on automated pipelines, the practical question is whether discovery is tightly scoped to trusted paths and explicit conventions. That is why secure build and verification guidance such as the OWASP Web Security Testing Guide is useful here: it reinforces the idea that security testing only works when the target set is well-defined. Broader control frameworks such as NIST SP 800-53 Rev 5 Security and Privacy Controls and NIST Cybersecurity Framework 2.0 also support the underlying discipline of controlled, monitored, and validated change in the software pipeline.

Risk and Threat Considerations

Overbroad test discovery can turn repository structure into an execution path, which creates exposure when unreviewed files are reachable by the test runner. The risk is not limited to accidental failures, because a tampered fixture, helper, or nested script can be executed during routine validation and blend into ordinary build activity.

Failure mechanism: Recursive matching, weak exclusions, or pattern drift cause the runner to include hidden or non-test files, then execute them with the same trust granted to legitimate tests.

Impact: The pipeline may leak secrets, corrupt test results, mask malicious changes, or provide an attacker with a low-friction execution path inside the build environment.

Practitioner Guidance

What to watch for: Keep discovery rules explicit, narrow, and version-controlled, especially when recursive scanning is enabled. Treat any expansion in the set of discovered tests as a change that deserves review, not as a harmless convenience improvement.

Governance implication: Owners of the build system should define which directories, file types, and naming conventions are eligible for discovery, then validate that the runner excludes generated content, hidden paths, and non-test code by design rather than by accident.

Related resources from NHI Mgmt Group

• How should security teams test APIs when endpoint discovery is incomplete?
• What breaks when hidden directories are included in test discovery?
• How should security teams test machine learning-based data discovery before relying on it in production?
• Why do AI agents that test internal networks and Active Directory need stronger access controls than external discovery tools?