Self-improving coding agents still fail at sustained vigilance

At a glance

What this is: This is ngrok’s analysis of a self-improving coding agent, bmo, and the key finding is that structured scaffolding mattered more than continual introspection.

Why it matters: It matters because agentic AI governance depends on knowing when behaviour is observable, when it is merely inferred, and when control must move from prompt logic to runtime boundaries.

👉 Read ngrok's full analysis of bmo and agentic self-improvement

Context

Self-improving coding agents create a governance problem that looks simple at first and quickly becomes architectural. The issue is not whether the agent can use tools, but whether it can reliably recognise friction, avoid deferring decisions, and apply the right behaviour at the right moment across sessions. In agentic systems, the gap between knowing and doing is where control failures begin, especially when tool access and decision-making are bundled together.

For identity and access programmes, this is an agentic AI governance problem with a genuine NHI angle. A coding agent that can choose tools, rewrite its own harness, and carry state across sessions is behaving like a non-human identity with delegated authority, so the operational question becomes how to constrain that authority over time rather than assuming better prompting will solve the issue.

Key questions

Q: How should security teams govern AI coding assistants that can execute commands?

A: Treat them as delegated non-human identities with bounded execution authority. Require human approval for destructive commands, keep command scopes narrow, and log every tool action. The key control question is not whether the assistant is helpful, but whether it can be prevented from acting outside intended scope when prompts, context, or rules are manipulated.

Q: Why do self-improving agents still miss recurring mistakes?

A: Because introspection is not the same as sustained vigilance. A model can analyse what happened after a task, but it is far less reliable at noticing patterns in real time while also completing work. That is why structured reflection and event-driven logging outperform vague instructions to “learn” from mistakes.

Q: What breaks when an agent has too many tools and too much choice?

A: Reliability drops because every extra tool increases the decision space, the number of failure paths, and the likelihood of inconsistent use. Narrow, safer tools with validation and defaults usually outperform broad command access for common tasks. The practical problem is not capability shortage, but control sprawl.

Q: How do you know if an agent harness is actually working?

A: You should see repeatable behaviour at the right moments: tool creation when friction appears, reflection at the end of each session, and fewer deferred tasks over time. If the agent keeps punting work into maintenance buckets, the harness is not governing behaviour, only describing it.

Technical breakdown

Why agentic coding loops drift without hard triggers

Agentic coding workflows are vulnerable to drift because the model must decide not only what to do, but when to change its own behaviour. If tool creation, learning capture, and reflection are all optional, the agent will usually fall back to the most probable next action in context, not the most secure or efficient one. That is why broad instructions such as “improve yourself when needed” underperform compared with explicit triggers. In practice, the control problem is less about intelligence and more about whether the harness reduces ambiguity enough for the model to act consistently.

Practical implication: define explicit runtime triggers for tool creation, reflection, and escalation instead of relying on free-form self-judgment.

How harness design shapes agent privilege and decision space

The article shows that fewer, safer tools can improve reliability because each additional tool expands the decision space and the chance of failure. A narrow harness that offers controlled functions, existence checks, and safer defaults reduces error-prone branching compared with unrestricted shell access. From an identity perspective, this is a privilege design issue: the agent’s authority is not just what it can access, but how many ways it can misuse that access. In other words, the harness becomes the policy boundary for an NHI-like workload identity.

Practical implication: prefer constrained toolsets with explicit guardrails over open-ended command execution for routine agent actions.

Why self-reflection is weaker than event-driven learning capture

Self-reflection works best when it is time-bound and structured. The post shows that a reflection template executed every session produced better results than an event-capture skill that depended on the model noticing its own mistakes in real time. That difference matters because continuous vigilance is a brittle assumption in LLM systems. The model can often analyse what happened after the fact, but it is much less reliable at interrupting itself mid-task to record lessons or alter behaviour. Governance should treat reflection as an orchestrated control, not an emergent property.

Practical implication: build scheduled review points and logging hooks into the agent workflow rather than expecting the model to self-monitor continuously.

NHI Mgmt Group analysis

Runtime self-improvement is a governance pattern, not a capability story. The post makes clear that the real question is not whether an agent can modify itself, but whether those modifications happen under controlled conditions. Without deterministic triggers, self-improvement becomes inconsistent and easy to defer. For agentic AI programmes, that means governance must focus on when the system may change behaviour, not just what the model can do.

Self-reflection only becomes useful when the control plane, not the model, owns the cadence. The reflection template worked because it was invoked at a predictable point in time. That is a stronger pattern than hoping the model will notice its own shortcomings during active work. Practitioners should treat this as a lesson in operating model-based systems: the schedule, logging, and review mechanics are part of the control design.

Agentic coding systems are effectively non-human identities with delegated operational authority. Once an agent can run commands, reload tools, and persist learning across sessions, it needs lifecycle controls that resemble NHI governance. That includes scope boundaries, approval points, and a clear limit on how much self-modification is allowed without human oversight. The practitioner conclusion is straightforward: if the agent has runtime discretion, it also needs governed identity and privilege.

Deferral is a failure mode created by structure, not intent. The article’s “battery change” loop created a bucket for postponed work, and that bucket made deferral more likely. That is a useful named concept for the field: deferral gravity, meaning the tendency of a system to push unresolved work into the easiest available future state. Teams should assume similar gravity will appear anywhere maintenance, reflection, and execution share the same context.

Better prompts cannot compensate for weak system boundaries. The article’s core lesson is that prompting can shape behaviour, but it cannot reliably create persistent vigilance or self-correction. That supports a broader AI governance view: durable control comes from harness design, state management, and explicit operational limits. For practitioners, the conclusion is to engineer the boundary conditions first and treat prompt quality as a secondary control.

What this signals

Deferral gravity: once an agent has an obvious place to park unresolved work, it will often take that path unless the runtime makes the safer choice the easiest one. That means governance for agentic systems must focus on friction points, not just instructions, because the easiest continuation in context usually wins. Practitioners should design for interruption, review, and controlled escalation.

The broader signal for identity teams is that agentic workflows are becoming operational identities with persistent context, delegated tools, and limited self-awareness. That pushes the programme toward lifecycle thinking: who can change the agent, when can it change itself, and what records prove that change was controlled. The control model should look closer to governed NHI lifecycle management than to one-off prompt tuning.

For practitioners

• Define hard triggers for agent self-improvement Require explicit conditions for tool creation, reflection, and backlog handling so the agent cannot defer important work into a later session. Use runtime prompts or orchestration logic to force action when friction is detected, rather than leaving the decision to the model.
• Reduce the agent’s decision space Replace broad shell-style access with safer, task-specific tools that include checks for file existence, command validation, and controlled outputs. This lowers failure rates because the agent has fewer paths to choose from during routine work.
• Separate reflection from execution Run session-end reflection as a scheduled control with fixed questions and persistent logging, then review the results outside the active task loop. That keeps learning capture from competing with delivery and makes missed lessons visible.
• Treat agent harnesses as identity boundaries Map each tool and skill to an explicit authority boundary, then review whether the agent can rewrite or extend that authority without oversight. This is especially important where the agent can act across multiple sessions or systems.

Key takeaways

• Self-improving coding agents do not become reliable simply because they can introspect; they become reliable when the harness forces the right behaviour at the right time.
• Tool sprawl and deferred work are control problems, not model quirks, because broad authority expands failure paths faster than it improves performance.
• Agentic systems that persist across sessions need NHI-style lifecycle governance, including explicit scope, controlled modification, and accountable review.

Key terms

• Agentic Harness: The orchestration layer that lets a model choose tools, execute actions, and repeat a task loop until a goal is met. In practice, it is what turns a model from a responder into an actor, so identity and privilege controls must govern the harness, not just the model.
• Deferral Gravity: The tendency for unresolved work to get pushed into a later maintenance cycle or reflection step because that bucket feels safer than handling friction immediately. In agentic systems, this becomes a structural failure mode when the runtime gives the model an easy place to postpone decisions.
• Learning Event Capture: A structured method for recording corrections, preferences, and repeated patterns while a session is still active or immediately after it ends. It turns informal feedback into durable memory, but it only works when the trigger is explicit enough that the system does not have to infer whether learning occurred.
• Runtime Self-Modification: The act of changing an agent’s tools, prompts, or operating rules while the system is in use. This can improve performance, but it also introduces governance risk because changes to authority, behaviour, or scope need review, logging, and rollback discipline to remain trustworthy.

What's in the full article

ngrok's full post covers the operational detail this post intentionally leaves for the source:

• The complete bmo harness design, including the tool model and session loop structure used to drive self-improvement.
• The full reflection template and learning-event-capture logic that shows how the agent was prompted to record lessons.
• The maintenance-pass workflow and backlog mechanics that explain why deferral kept appearing in practice.
• The detailed performance table for tools and skills across sessions, including where the agent improved and where it regressed.

👉 ngrok's full post includes the harness design, session mechanics, and the detailed lessons from two weeks of use.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, and secrets management. It helps security and identity practitioners build the controls needed for delegated systems that can act, learn, and persist across sessions.