Should teams prioritise real-device testing or AI path discovery first?

Why real-device execution has to come before AI path discovery

Teams should start with the test engine, not the path-finding layer. If a system cannot authenticate reliably, preserve session state, and complete multi-step workflows on a real device, then AI-driven path discovery is only navigating around an execution gap. The result is faster scanning, but not stronger coverage.

Real-device testing exposes whether the app actually behaves correctly under the same browser, OS, timing, permission, and state conditions users face. That matters because many failures only appear when the test can carry context across steps, handle dynamic UI changes, and prove that a flow reaches a meaningful end state rather than a partial page transition.

AI path discovery is still useful, especially for expanding exploration, finding alternate routes, or identifying screens and branches a human tester might miss. But it is a multiplier, not a substitute. When the underlying runner is shallow, AI can create the appearance of breadth while leaving the hard part, authenticated stateful execution, unvalidated.

What each layer is actually good at

Real-device testing is the verification layer. It checks whether the application can be used end to end under realistic conditions, including login, redirects, multi-factor steps, session expiry, storage, permissions, and back-and-forth navigation. It is the only layer that can tell you whether a journey is not just discovered, but truly executable.

AI path discovery is the exploration layer. It can help prioritise paths, infer likely routes from structure and behaviour, and reduce manual effort in large or complex applications. That makes it valuable for scaling test creation, but only after the engine can already execute deterministic flows with enough fidelity to trust the outputs.

The practical distinction is that discovery tells you where to try, while device execution tells you whether the path is real. In mature testing programmes, these should work together, but in the wrong order ai discovery can mask weak automation by making low-quality coverage look intelligent.

How to sequence the investment without overbuilding the wrong thing

Build the foundation first around reliable device-level execution, state handling, and workflow completion. Once that is stable, use AI to broaden reach, reduce authoring effort, and uncover untested branches. If you reverse the order, you usually end up tuning the explorer while the executor still cannot hold a logged-in session long enough to matter.

• Verify that the test harness can authenticate and remain authenticated long enough to complete the target flow.
• Confirm that context, cookies, tokens, and app state survive the steps that matter for the journey under test.
• Use AI discovery to expand coverage only after repeatable execution produces trustworthy pass and fail signals.
• Escalate any “found” path that cannot be replayed on a real device, because it is probably not a dependable test path yet.

Risk and Threat Considerations

The main risk is false confidence. If AI path discovery runs ahead of execution quality, teams can end up with a large set of apparently covered journeys that never actually complete under real conditions. That creates blind spots in regression, weakens release decisions, and can hide authentication, session, or state-handling defects.

Failure mechanism: a shallow scanner may enumerate routes, but without stable authenticated execution it cannot validate the transitions, context retention, or end-to-end business action that make a test meaningful.

Impact: organisations may believe they have broader coverage than they really do, while the most failure-prone flows remain unproven and defects escape into production.

Practitioner Guidance

What to prioritise: Start with the smallest set of real-device flows that prove login, context preservation, and completion of the highest-value user journeys. Those flows define whether later AI-discovered paths are actionable or just illustrative.

Decision rule: If a path cannot be replayed and observed on a real device with consistent state, treat it as discovery output, not test coverage. If it can be executed reliably, then use AI to widen the search for adjacent paths and edge cases.

What practitioners underestimate: The hardest testing problem is often not finding more paths, but proving that a path remains valid across time, state changes, and authentication boundaries.

Practitioner takeaway: Real-device execution establishes trust in the test signal; AI adds leverage only after that trust exists.

Related resources from NHI Mgmt Group

• How should security teams evaluate AI penetration testing tools for real-world coverage in developer-first environments?
• Should security teams prioritise AI inventory or adversarial testing first?
• Should security teams prioritise shadow AI discovery or policy writing first?
• Should teams prioritise runtime validation or cloud discovery for AI security first?