AI system iteration: why harness engineering beats model tuning

TL;DR: Most AI teams now get better results by improving prompts, retrieval, tools, evals, and traces instead of retraining models, while only a small group with heavy ML infrastructure still pursues continuous reinforcement learning, according to Arize. The shift matters because production AI governance increasingly depends on the harness around the model, not just the model weights.

NHIMG editorial — based on content published by Arize: The end of fine-tuning: Why evals, context, and traces matter more

Questions worth separating out

Q: How should security teams govern AI models that can call tools and access data?

A: Security teams should govern AI models as non-human identities with named owners, limited scope, short-lived credentials, and continuous authorization.

Q: Why do evals matter more than model benchmarks in production AI?

A: Benchmarks measure models in isolation, but production systems depend on prompts, retrieval, tools, memory, and routing.

Q: What do organisations get wrong about fine-tuning for AI products?

A: They often treat fine-tuning as the main route to better behaviour when the easier gain is in the harness around the model.

Practitioner guidance

• Instrument AI traces end to end Capture prompts, retrieved context, tool calls, and outputs so you can reconstruct how the system reached a decision or action.
• Scope AI tool access like non-human identity Assign each AI workflow the minimum set of tools, APIs, and data sources it needs, then review that access as a lifecycle asset rather than a static app setting.
• Use eval gates before deployment Require release approval when evals show unsafe tool use, context leakage, or inconsistent outcomes across representative sessions and traces.

What's in the full article

Arize's full blog post covers the operational detail this post intentionally leaves for the source:

• The article’s model-by-model examples of how the top 1% of AI teams structure continuous RL and harness co-optimisation.
• The full discussion of prompt engineering, context engineering, and harness engineering as distinct iteration layers.
• The article’s explanation of why evals need to score traces, sessions, and trajectories instead of isolated model responses.
• The source’s examples of how production data is converted into release gates, routing logic, and feedback loops.

👉 Read Arize’s analysis of why evals, context, and traces matter more than fine-tuning →

AI system iteration: why harness engineering beats model tuning?

Explore further

View Full Forum →  |  NHI Foundation Course →