Agentic AI for flaky tests: what Kong’s workflow reveals

At a glance

What this is: Kong describes an agentic workflow that identifies, fixes and verifies flaky tests in its Gateway CI pipeline, showing that autonomous task loops can reduce the time and attention needed to resolve intermittent failures.

Why it matters: This matters because teams are starting to use agentic systems for production engineering work, and identity and governance models have to account for autonomous execution, scoped tool use and context boundaries rather than only human-operated workflows.

Context

Flaky tests are intermittent failures that pass on rerun, which makes them expensive to diagnose and even more expensive to verify. In Kong's case, the problem sits inside a large CI system where long test runs and aggressive sharding already create pressure on engineering time and pipeline capacity.

The governance question is no longer whether automation can assist developers. It is whether an agentic workflow can be trusted to operate inside a bounded engineering process without losing context, inventing fixes, or masking the real cause of the failure. For teams exploring agentic AI, this is a practical example of runtime autonomy applied to a narrow but high-value task.

Key questions

Q: What breaks when an agentic workflow is given too much context during debugging?

A: The agent starts treating old attempts as current evidence, which can dilute hypothesis quality and slow down resolution. Agentic debugging works best when each run starts with a tight evidence set, carries forward only a summary of prior attempts and stops when the verification rule is satisfied.

Q: Why does repeated verification matter more than a single successful rerun?

A: Because flaky behaviour is intermittent, a one-off success proves very little. Repeated verification checks whether the fix holds under the same conditions that previously caused failure, which is the only reliable way to separate a true fix from a temporary pass.

Q: How should teams decide which engineering tasks are suitable for agentic automation?

A: Choose tasks that can be bounded, measured and repeated without ambiguity. If the work can be expressed as a stable loop with clear inputs, outputs and success criteria, an agent can often handle it; if the problem depends on broad interpretation or shifting goals, it should stay human-led.

Q: Should human review disappear once an agent has prepared a fix?

A: No. Human review still matters at the point where the change is promoted into the shared codebase, especially when the workflow is autonomous during investigation and verification. The agent can accelerate diagnosis, but governance should still control the final merge decision.

Technical breakdown

Identify-fix-verify loops in agentic CI workflows

Kong's workflow breaks the work into a top-level orchestrator, a flake-fixer subagent and a flake-verifier subagent. The orchestrator pulls failure logs, seeds the investigation and repeats the cycle until a run is clean enough to open a pull request. This is not simple automation. The agent is choosing hypotheses, reading log evidence, creating branches and deciding when verification is strong enough to proceed. That makes the workflow a useful example of agentic task design, where runtime judgment matters more than scripted repetition.

Practical implication: treat each autonomous stage as a governed task boundary with explicit inputs, outputs and stop conditions.

Why context management determines whether the agent succeeds

The article shows that long-running agentic work fails when context becomes too broad. Kong deliberately keeps the flake-fixer narrow, gives it one analysis at a time, and carries forward only a summary into the next cycle. That preserves signal, avoids context-window bloat and reduces the chance that earlier dead ends shape later conclusions. In agentic systems, context is not just a model-input problem. It is a governance control that determines whether the system can stay focused long enough to produce a reliable result.

Practical implication: constrain what an agent can carry across iterations and reset state where prior context is no longer decision-useful.

Verification is a control, not a finishing step

Kong's loop does not treat verification as a quick human sign-off. It repeatedly reruns the affected test, filters unrelated noise and only promotes the fix after a clean streak. That matters because flaky behaviour is intermittent by definition, so a single successful rerun says very little. The article also notes that the workflow surfaced genuine bugs in code that humans had missed, which suggests the verification phase is doing more than confirming the fix. It is acting as a discovery control inside the development process.

Practical implication: design agentic verification to prove repeatability, not just to confirm a one-off success.

NHI Mgmt Group analysis

Runtime autonomy changes engineering workflows by collapsing the human babysitting model. This article is not about identity security in the narrow sense, but it is highly relevant to the governance of autonomous systems because the agent is making decisions inside a bounded software task without waiting for human approval between steps. That shifts the control problem from manual oversight to delegated execution design. Practitioners should read this as evidence that autonomous behaviour can be operationally useful only when the task boundary is narrow and the verification rule is explicit.

Context is a governance asset in agentic systems, not just a model input. Kong's approach works because each flake-fixer starts from a focused seed file and carries forward only the summary of prior attempts. That is a practical answer to the problem of agent drift, where too much historical detail can dilute current reasoning. For teams building or reviewing agentic workflows, the key question is not how much context an agent can hold, but how much context it should hold to remain accountable and effective.

Verification loops are becoming the real control plane for agentic work. In this workflow, the system only advances when repeated checks show the failure has truly stopped recurring. That matters because autonomous systems can appear successful while still missing intermittent defects. The category is moving toward proof-based execution, where the organisation trusts the process only after the agent can demonstrate repeatable outcomes.

Agentic debugging reveals the difference between automation and autonomy. The workflow can launch investigations, create branches and decide whether the evidence is enough to continue, which goes beyond static automation. At the same time, it still depends on a human code review before merge, so the article also shows a useful boundary: autonomy can accelerate technical work without removing human governance from final approval. That distinction is central to how teams should classify emerging agentic systems.

Flaky-test remediation is a useful named concept for agentic engineering. It shows how an agent can be assigned a narrow investigative loop where success is measurable, context is bounded and repeated verification is built into the task. That pattern is likely to spread to other repetitive engineering problems, but only where the task can be decomposed into a stable identify-fix-verify sequence. Practitioners should treat it as a design pattern for controlled autonomy, not a generic promise of agentic productivity.

From our research library:

• 88% of organisations have embedded AI agents in their workflows, according to KPMG's 2026 report.
• Read next: Agentic AI Identity Maturity Model

What this signals

Agentic engineering works best when the workflow is narrow enough to be governed as a task, not as a general assistant. The article shows that success depends on strict task framing, repeatable verification and minimal carry-forward context. That is the same design pressure security teams will face when autonomous systems start operating inside CI, incident response and remediation pipelines.

Autonomous loops will expose whether organisations are still relying on static oversight patterns. According to the 2026 Infrastructure Identity Survey, 67% of organisations still rely heavily on static credentials despite the risks they pose to agentic AI deployments. That gap matters because agent-timed work needs issuance, verification and revocation patterns that match runtime decision speed, not human scheduling.

For practitioners

• Define bounded agentic task loops Use a narrow identify-fix-verify pattern when assigning autonomous work so the agent has clear inputs, clear exit criteria and a single outcome to prove before escalation.
• Separate investigation from verification Give the investigation step enough depth to form a hypothesis, then use a different verification step to test recurrence rather than trusting the first successful rerun.
• Keep carry-forward context minimal Pass only the summary of previous attempts into the next agent run so earlier dead ends do not crowd out fresh analysis or inflate the context window.
• Preserve human approval at merge time Let the agent prepare the fix, but keep human review as the final governance gate before code is merged into the main branch.

Key takeaways

• Agentic workflows can materially reduce the time needed to resolve flaky tests when they are tightly bounded and verified in stages.
• The key constraint is not model intelligence alone but how well the workflow preserves context, isolates tasks and proves repeatability.
• Human review remains important even when autonomous agents handle investigation and verification, because promotion into the main codebase is still a governance decision.

Key terms

• Agentic workflow: An agentic workflow is a sequence of tasks executed by an AI agent with some level of tool access and decision authority. In security terms, the workflow matters because it can span multiple systems, identities, and permissions, which makes attribution and revocation harder than with ordinary automation.
• Flaky Test: A flaky test is a test that fails intermittently without a stable code change explaining the failure. The underlying problem is often nondeterminism in timing, ordering, or shared state, which makes diagnosis expensive and verification difficult.
• Context Window: The context window is the text a model receives at one time, including prompts, retrieved documents, and conversation history. Security teams care about it because it becomes the practical boundary between trusted instructions and untrusted content, especially when the application assembles that text automatically.
• Verification loop: The cycle in which an agent takes an action, sees the result, and adjusts its next move. In autonomous or semi-autonomous workflows, the loop is the real control boundary, because it determines how quickly an identity can iterate without human review.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.