Evals, context, and traces now matter more than fine-tuning

At a glance

What this is: This is Arize’s argument that AI product iteration has split between heavy model training for a small elite and harness engineering for everyone else.

Why it matters: It matters to IAM practitioners because AI governance now depends on tracing, evaluation, and control of the surrounding system, including how agents use tools, contexts, and feedback loops.

👉 Read Arize’s analysis of why evals, context, and traces matter more than fine-tuning

Context

AI product development has split into two operating models: a small number of teams still train models continuously, while most teams iterate through prompts, retrieval, tools, evals, and traces around the model. That shift is not just a model-selection issue. It changes how security, governance, and accountability work when AI systems act through external tools and retained context.

For identity and access programmes, the implication is direct. AI systems that call tools, retain memory, and act on feedback behave more like governed software systems than static applications, which means access boundaries, auditability, and traceability matter as much as model quality. The article is most useful where it shows that the control plane has moved outward from weights into the harness.

This is a typical pattern for mature AI adoption: the organisations with the most operational maturity can justify model training, but the majority get more value from making the surrounding system safer, more observable, and easier to govern.

Key questions

Q: How should security teams govern AI models that can call tools and access data?

A: Security teams should govern AI models as non-human identities with named owners, limited scope, short-lived credentials, and continuous authorization. The critical shift is to treat every tool call, data read, and update path as a privileged action that can be logged, revalidated, and revoked. Without that discipline, model risk becomes identity risk.

Q: Why do evals matter more than model benchmarks in production AI?

A: Benchmarks measure models in isolation, but production systems depend on prompts, retrieval, tools, memory, and routing. Evals show whether the whole system behaves safely under real conditions. They also create the feedback loop that drives release decisions, exception handling, and governance review.

Q: What do organisations get wrong about fine-tuning for AI products?

A: They often treat fine-tuning as the main route to better behaviour when the easier gain is in the harness around the model. Most teams need stronger retrieval, clearer prompts, safer tool scopes, and repeatable evals before they need new weights. Fine-tuning is a niche capability, not a default fix.

Q: How do AI traces help with access control and accountability?

A: Traces show which context the system used, which permissions it exercised, and which actions followed. That makes them essential for investigating misuse, proving scope, and deciding whether an AI workflow exceeded its intended authority. Without traces, access control becomes guesswork after the fact.

Technical breakdown

Why harness engineering has replaced model-only iteration

Harness engineering treats the AI application as a system made up of prompts, retrieval, tools, memory, routing, retries, and verification gates. The model is only one component. In practice, most product failures come from how those parts interact, not from the base model alone. That is why teams can often improve behaviour without retraining weights. A well-designed harness also survives model swaps and vendor deprecations better than a tuned artifact, because the operational logic sits outside the model lifecycle.

Practical implication: treat prompts, tools, and guardrails as governable production assets, not disposable experiments.

Why evals are the control surface for AI governance

Evals are no longer just benchmark scores against static datasets. In production, they measure system behaviour across spans, traces, sessions, and trajectories, then route failures into review, alerts, or CI gates. That makes evals a governance control, not just a research metric. If a system uses retrieval, external tools, or memory, the relevant question is whether the whole workflow behaves safely and consistently under real conditions. Without that feedback loop, changes to prompts or tooling become guesswork.

Practical implication: build evals that score end-to-end system behaviour and connect failures to release and review decisions.

How traces expose the identity and access side of agentic AI

Traces show which context the system used, which tools it called, and what sequence of actions followed. For identity teams, that is where authorisation, delegation, and accountability become visible. If an AI system can access internal tools, external APIs, or sensitive data, the trace is the only practical way to reconstruct whether it stayed within its permitted scope. This is where agentic AI intersects with IAM and NHI governance: every tool call is an access decision that should be observable and reviewable.

Practical implication: require trace-level visibility for any AI system that can invoke tools or reach sensitive data.

Threat narrative

Attacker objective: The attacker aims to exploit the AI system’s delegated access path so it can expose data, misuse tools, or trigger actions beyond intended scope.

1. Entry occurs through publicly exposed credentials, permissive tool access, or overly broad context retrieval that lets an AI system reach resources it should not touch.
2. Escalation follows when the model is allowed to chain tool calls, reuse context, or act on weakly scoped permissions without traceable approval boundaries.
3. Impact appears as unsafe external actions, data leakage, or decision corruption, because the system’s surrounding harness failed to constrain what the model could access or do.

NHI Mgmt Group analysis

Harness engineering is now the real security boundary for AI systems. The article’s central point is that value has shifted from model weights to the surrounding system of prompts, retrieval, tools, traces, and evals. That matters to identity governance because access decisions now happen inside the harness, not just at login. Practitioners should treat the harness as the control plane for AI behaviour.

Tracing and evaluation create the missing accountability layer for agentic AI. A model without traces can produce output, but a system with tool access needs provenance, replayability, and outcome scoring. That is where NHI and IAM concerns surface: AI systems increasingly behave like non-human actors with delegated permissions. Organisations that cannot explain which data, tools, or contexts drove an action will struggle to govern it.

Fine-tuning is becoming a niche capability, not a default governance strategy. The article shows that only a small population of well-resourced teams can justify continuous model training. For everyone else, the practical pattern is operational control through retrieval, prompts, guardrails, and eval loops. That does not reduce governance needs. It shifts them into change control, access control, and traceability across the full AI stack.

Agentic AI creates a named concept: the harness trust gap. This is the gap between what teams assume the model can safely do and what the surrounding harness actually permits through tools, memory, and context. When that gap is unmanaged, the AI system becomes a delegated identity with unclear bounds. Practitioners should close the gap with explicit permissions, trace review, and release gates.

What this signals

Harness trust gap: AI teams now need to govern the space between model capability and permitted system behaviour. That means traceability, scoped tool access, and release gating become core controls for any workflow that can retrieve data or invoke actions.

The practical signal for identity programmes is that AI agents and automated workflows are becoming policy-bearing systems, not just consumers of API keys. This is where NIST SP 800-53 Rev 5 Security and Privacy Controls and trace-based review practices intersect with AI governance, especially where tool calls create privileged actions.

Teams that wait for a model-training problem before investing in governance will miss the real control point. The better near-term move is to harden the harness, define access boundaries for every AI workflow, and use traces to make those boundaries auditable.

For practitioners

• Instrument AI traces end to end Capture prompts, retrieved context, tool calls, and outputs so you can reconstruct how the system reached a decision or action. This should be mandatory for any workflow that touches sensitive data or production systems.
• Scope AI tool access like non-human identity Assign each AI workflow the minimum set of tools, APIs, and data sources it needs, then review that access as a lifecycle asset rather than a static app setting.
• Use eval gates before deployment Require release approval when evals show unsafe tool use, context leakage, or inconsistent outcomes across representative sessions and traces.
• Prefer harness changes before retraining Fix prompt design, retrieval boundaries, and verification logic first, then consider model training only when the production loop proves those controls are insufficient.

Key takeaways

• Most AI teams should optimise the harness around the model before they consider retraining the model itself.
• Evals and traces are becoming governance controls because they expose how AI systems actually use context, tools, and permissions.
• AI systems with delegated access need identity-style boundaries, or their behaviour will outrun the controls built for static software.

Key terms

• Harness Engineering: Harness engineering is the set of systems around a model that make it reliable in production. It includes orchestration, retries, guardrails, observability, evaluation, and fallbacks, all of which determine whether the agent can be trusted at enterprise scale.
• Evaluation Harness: An evaluation harness is the test environment, scripts, permissions, and network setup used to assess a model or system before release. Its security matters because the harness is often the difference between harmless testing and a real-world compromise path if it is allowed to reach external systems.
• Context Engineering: The practice of selecting, curating, and delivering the information an AI system uses at runtime. In agentic environments, context engineering is a security function because the quality, provenance, and trust level of the inputs directly shape the system’s actions and outputs.
• Identity Traceability: Identity traceability is the ability to link each action back to a specific identity, authorisation path, and time window. It is essential when humans, service accounts, and AI agents all operate in the same environment and auditors need a defensible record.

What's in the full article

Arize's full blog post covers the operational detail this post intentionally leaves for the source:

• The article’s model-by-model examples of how the top 1% of AI teams structure continuous RL and harness co-optimisation.
• The full discussion of prompt engineering, context engineering, and harness engineering as distinct iteration layers.
• The article’s explanation of why evals need to score traces, sessions, and trajectories instead of isolated model responses.
• The source’s examples of how production data is converted into release gates, routing logic, and feedback loops.

👉 Arize’s full post covers the split between model training teams and harness-first teams in more operational detail

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management in the context of modern access control. It is designed for practitioners who need to govern delegated systems, not just human users.