Prompt learning for Claude Code shows prompt-level gains

At a glance

What this is: Arize’s analysis shows that prompt learning can materially improve Claude Code performance using only system-prompt changes, LLM evals, and iterative optimisation.

Why it matters: For IAM, NHI, and agentic AI teams, this matters because prompt-level control is now part of the governance boundary for software that can act, decide, and change code.

By the numbers:

• Arize reports a 5.19% test accuracy gain on the by-repo split after prompt optimisation.
• Arize reports a 10.87% improvement on the in-repo test after prompt optimisation.

👉 Read Arize’s analysis of Prompt Learning on Claude Code

Context

Prompt learning moves part of the control problem from model weights to instructions, which makes the system prompt a governance boundary rather than a convenience layer. In agentic AI settings, that matters because behaviour can change materially without any change to the underlying model, tools, or runtime.

Claude Code is a useful case study because it sits close to software production workflows where small prompt changes can alter patch quality, repo fit, and error handling. For identity and AI governance teams, the key question is not whether the agent can code, but how its instruction surface is controlled, reviewed, and separated by environment.

The article’s starting position is typical for modern agentic systems: performance tuning through prompts is increasingly practical, but the control implications are still underdeveloped.

Key questions

Q: How should security teams govern prompt changes in AI agent systems?

A: Treat prompt updates as production changes that can alter access, not just behaviour. Put them through approval, logging, testing, and rollback controls, especially when prompts influence retrieval, tool use, or data exposure. The right question is whether the change can expand what the agent can do with existing identities, tokens, or secrets.

Q: Why do repository-specific prompts create governance risk for AI agents?

A: Because they can improve local performance while making behaviour less portable and less predictable elsewhere. A prompt that fits one codebase well may encode assumptions, conventions, or shortcuts that fail in another environment, so teams need separate baselines and explicit approval for local tuning.

Q: What do organisations get wrong about prompt optimisation for AI agents?

A: They often treat prompt optimisation as a harmless productivity tweak rather than a control change. In practice, prompts can alter how an agent interprets tasks, applies constraints, and handles exceptions, so prompt optimisation should be governed with the same discipline as other production settings.

Q: How should teams evaluate whether a coding agent is actually improving?

A: Do not rely on a single pass rate. Combine task success metrics with explanation-rich evaluations that identify why the agent failed or succeeded, then compare results across shared and repository-specific environments to detect brittle improvements and overfitting.

Technical breakdown

How prompt learning changes agent behaviour without retraining

Prompt learning is an optimisation loop that changes the instructions given to an agent rather than its model weights. In this case, Claude Code was evaluated against SWE Bench Lite issues, and the optimiser used feedback from LLM-based evaluation to adjust the system prompt. That means the behaviour shift comes from better task framing, repo guidance, and failure correction, not from fine-tuning or architecture changes. For agentic systems, this creates a distinct governance surface because the prompt can encode constraints, preferences, and operating rules that shape every downstream action.

Practical implication: treat system prompts as governed configuration, with review, versioning, and change control.

Why LLM evals are stronger than scalar reward signals

A scalar pass or fail score tells you whether a patch worked, but not why it failed. Arize’s approach uses LLM feedback to identify conceptual errors, missing context, and repo-specific mistakes, then feeds those diagnostics back into the optimiser. That is closer to incident analysis than simple scoring. For AI governance, this matters because the quality of feedback determines whether the system learns stable operational patterns or just overfits to noisy success indicators. The method is especially relevant when agents operate in codebases where conventions and edge cases drive outcomes.

Practical implication: build evaluation pipelines that explain failure modes, not just record pass rates.

What repo-specific prompt tuning means for AI agent governance

The article’s split-by-repository and within-repository testing shows two distinct governance outcomes. General prompt improvement suggests reusable controls across codebases, while repo-specific tuning intentionally captures local conventions, expectations, and workflows. In practice, that creates a policy question: when does local optimisation become uncontrolled specialisation? For organisations using coding agents, the same mechanism that improves productivity can also create hidden behavioural drift across teams, repos, or environments if prompt variants are not tracked and approved.

Practical implication: maintain separate prompt baselines for shared use cases and repository-specific use cases.

NHI Mgmt Group analysis

Prompt governance is becoming a security control, not a tuning preference. The article shows that changing only the system prompt can move an advanced coding agent by measurable amounts. That means prompt content, prompt ownership, and prompt change review now belong in the control stack alongside access, logging, and release management. For agentic AI programmes, the relevant question is who can alter behaviour, under what approval model, and with what evidence of impact.

Repository-specific optimisation creates a new form of behavioural drift. The in-repo gains are operationally useful, but they also show how quickly an agent can become tightly adapted to one environment’s conventions. That is a named concept worth tracking: prompt locality risk: when a prompt becomes so tailored to one codebase that it behaves unpredictably elsewhere. Practitioners should treat local tuning as a governed exception, not a default.

LLM evals are a governance asset because they expose failure modes, not just outcomes. A pass rate tells you whether the agent succeeded; an explanation tells you whether it misunderstood context, missed edge cases, or relied on brittle assumptions. That makes evaluation traces useful for auditability and model-risk review. In AI governance terms, the value is not just optimisation but traceable reasoning about why a prompt changed.

Claude Code is a proxy for the broader identity problem of software that can act on its own instructions. Once an agent can consume repository context, adapt its behaviour, and generate code, its prompt becomes part of its operational identity. That overlaps with NHIs because the agent behaves like a governed machine actor with scoped permissions and persistent instruction memory. The discipline here is to manage agent identity, instruction surfaces, and lifecycle controls together.

Performance gains without retraining do not reduce governance burden. The article shows that system-prompt optimisation can deliver useful results without touching model weights, but that also means the highest-leverage control may sit in the least visible place. Organisations should not confuse “no model change” with “low risk.” The practitioner conclusion is simple: prompt changes need the same review discipline as production configuration changes.

What this signals

Prompt learning will push more organisations to treat agent prompts as governed artefacts, especially where coding agents can influence repositories, tests, or deployment paths. The governance lesson is that behavioural changes can arrive through configuration, not code, which means security teams need evidence trails for instruction changes as well as execution logs. The control model is moving closer to machine identity management than traditional model tuning. For background on the operational trust gap, see the Ultimate Guide to NHIs.

Prompt locality risk: this is the practical problem created when an agent becomes highly tuned to one repository’s norms but loses portability across the rest of the estate. That same pattern can make exception handling harder to audit, because the prompt becomes the hidden policy layer. Organisations using coding agents should expect more demand for prompt inventories, approval workflows, and rollback capability. For a related AI security lens, compare this with OWASP Agentic AI Top 10.

For practitioners

• Version and approve system prompts Treat Claude Code prompts and CLAUDE.md files as production configuration. Store prompt versions, require review for changes, and tie each release to a business owner so behavioural drift is traceable.
• Separate baseline and repo-specific prompt policies Maintain a shared prompt baseline for general coding tasks and a separate, approved layer for repository-specific instructions. This reduces accidental leakage of local conventions into unrelated codebases.
• Use explanatory evals in prompt optimisation loops Pair pass or fail metrics with LLM-written failure analysis so the optimiser learns from root causes such as missing context, wrong assumptions, or brittle repository knowledge.
• Map coding agents to scoped machine identity controls Apply least privilege, repository scoping, and change logging to coding agents that can write code or trigger tools. The agent’s prompt and permissions should be managed together, not as separate concerns.

Key takeaways

• Prompt learning can materially improve a coding agent without retraining the model, which makes the prompt itself a governance boundary.
• LLM evals are valuable because they surface failure modes, not just success rates, and that improves both optimisation and auditability.
• Repository-specific tuning can be useful, but without prompt versioning and approval it creates behavioural drift that teams will struggle to detect.

Key terms

• Prompt Learning: Prompt Learning is the practice of improving an agent by iteratively refining the instructions it receives rather than changing model weights. It uses evaluation feedback to adjust prompt wording, structure, and constraints so the agent behaves better in a specific task or environment.
• Meta-prompting: Meta-prompting is the process of asking a language model to improve another prompt based on observed performance. In agent workflows, it turns prompt optimisation into a feedback loop where the model helps refine the instructions that govern behaviour.
• LLM Evaluation: LLM evaluation is the process of scoring model outputs against defined criteria so teams can judge quality before or after release. It typically uses test datasets, scorers, and review workflows to measure correctness, safety, relevance, and task success in a repeatable way.
• Prompt Locality Risk: Prompt locality risk is the chance that a prompt tuned for one repository, team, or workflow becomes brittle or misleading when reused elsewhere. It is a governance concern because the same optimisation that improves local performance can reduce portability and increase hidden behavioural drift.

What's in the full article

Arize’s full blog post covers the implementation detail this analysis intentionally leaves for the source:

• Step-by-step Prompt Learning workflow showing how the optimiser was run across Claude Code training examples
• Split-by-repository and in-repository setup details that explain how the evaluation avoided or encouraged specialisation
• Examples of the meta-prompting loop used to improve prompts from LLM feedback
• Operational notes on using Phoenix experiments to run agent evaluations and extract diffs

👉 The full Arize post covers the prompt optimisation loop, train-test splits, and result interpretation in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, IAM, machine identity security, and secrets management. It helps practitioners build the control discipline needed when software agents and service identities begin to act like operational actors.