Test Discovery

Expanded Definition

Test discovery is the build-time and run-time logic that identifies which files, modules, or packages should be executed as tests. In software pipelines, the term often sounds mundane, but it becomes a security boundary when recursive discovery walks into directories that were never intended to be treated as executable test inputs.

In NHI and agentic environments, the risk is not the test concept itself but the trust decision made by the toolchain. A discovery rule may accept naming patterns, file locations, or metadata and then hand those candidates to a runner with broad filesystem or network reach. That is why the surrounding governance matters: the behaviour should be reviewed alongside least privilege, repository hygiene, and pipeline trust assumptions described in the Top 10 NHI Issues and the NIST Cybersecurity Framework 2.0.

Definitions vary across build tools, because some treat discovery as a naming convention problem while others treat it as a file traversal problem. The most common misapplication is assuming recursive discovery is safe by default, which occurs when hidden directories, generated artifacts, or unreviewed vendor content are present in the repository.

Examples and Use Cases

Implementing test discovery rigorously often introduces coverage-versus-control tradeoffs, requiring organisations to weigh broad automated detection against the risk of executing unintended code paths.

• A CI job scans a repository for files matching a test prefix, then executes everything under nested folders, including copied sample code that was never reviewed as trusted project code.
• A monorepo uses discovery rules to pick up integration tests, but a hidden directory added by a tooling migration is also traversed and exposed to the runner.
• A security team compares discovery output against a fixed allowlist so only approved test paths run, reducing surprise execution in release pipelines.
• During pipeline hardening, engineers pair discovery rules with repository controls so that generated files, caches, and secrets-bearing folders are excluded before execution.

For a broader lifecycle lens on how tooling decisions affect trust boundaries, the NHI Lifecycle Management Guide is useful because the same discipline that governs onboarding and rotation also applies to what a build system is allowed to touch. In standards-based practice, test discovery should be read alongside the NIST Cybersecurity Framework 2.0 so that automated execution paths are not treated as harmless defaults.

Why It Matters in NHI Security

Test discovery matters because build and pipeline tooling often runs with elevated access to source, secrets, and deployment credentials, which means an overly broad discovery rule can become a route from untrusted files to privileged execution. In NHI security, this is especially relevant when service credentials or automation tokens sit near code artifacts that discovery engines can traverse.

The operational stakes are high. NHI Mgmt Group reports that 96% of organisations store secrets outside of secrets managers in vulnerable locations including code, config files, and CI/CD tools, and 79% have experienced secrets leaks. When discovery is not constrained, those locations are more likely to be executed, parsed, or otherwise surfaced during pipeline runs. The same issue is covered from a risk-management angle in the Ultimate Guide to NHIs — Key Challenges and Risks, which explains how visibility gaps and configuration drift expand exposure.

Organisations typically encounter the impact only after a pipeline has executed an unexpected file, at which point test discovery becomes operationally unavoidable to address.

Related resources from NHI Mgmt Group

• How should security teams test APIs when endpoint discovery is incomplete?
• Why is NHI discovery and inventory the primary goal of NHI security?
• Why is continuous discovery of AI agents important?
• When does secrets discovery become insufficient on its own?