What is the difference between running an eval and building a custom completion function?

Running an eval versus building a custom completion function

An eval and a custom completion function solve different problems inside the same testing loop. The eval is the measurement layer: it defines the benchmark, runs the cases, and records results. The custom completion function is the execution layer: it tells the eval framework how to get a result for each case when the default model call is not enough.

What the eval is responsible for

An eval is built to answer a measurement question, such as whether a model can follow instructions, complete a task, or meet a target quality bar. Its value comes from consistency, repeatability, and comparable scoring across runs. If you change the model, prompt, dataset, or scoring rule, the eval should still give you a reliable way to see what changed.

That means the eval should stay focused on case selection, input setup, scoring, and aggregation. It is not the place to hide extra business logic or task-specific orchestration. When an eval becomes overloaded with execution steps, it becomes harder to interpret because the score no longer reflects model performance alone.

What the custom completion function adds

A custom completion function exists when the benchmark needs more than a plain prompt and response. It can add preprocessing, external calls, chained steps, tool use, or postprocessing before the eval can score the outcome. In other words, it adapts the benchmark’s execution path so the test can represent the real task instead of only a simplified prompt completion.

This is especially useful when the task involves structured outputs, retrieval, calculations, or other operations that the base model call will not perform correctly on its own. The function becomes the bridge between the eval case and the actual workflow you want to test. That makes it part of the test harness, not the scoring logic.

How to decide which one you need

If your question is “How well does the model do this task as presented?”, the eval should do the heavy lifting and the completion path should stay simple. If your question is “How well does the model handle this task when the task requires extra operations?”, then a custom completion function is appropriate because it lets the benchmark reflect the real execution requirements.

The practical difference is therefore scope. The eval defines what success looks like; the completion function defines how to reach the point where success can be measured. Keeping that boundary clear makes your scores easier to trust, compare, and debug when they move.

Risk and Threat Considerations

When the completion function includes external operations, the eval can start measuring system behavior rather than model behavior. That creates a risk of hidden coupling, where failures in retrieval, tools, or preprocessing look like model quality problems even though the underlying issue is in the harness.

Failure mechanism: A custom completion function can introduce side effects, unstable dependencies, or inconsistent execution paths that change scores without changing the model itself.

Impact: Teams may optimize the wrong layer, compare runs unfairly, or accept a benchmark result that does not represent real model capability.

Practitioner Guidance

What to verify: Keep the scoring rule separate from the execution logic, and verify that the completion function is deterministic enough for repeated testing. If the benchmark needs tools or external steps, document those dependencies so future runs can be reproduced.

Common mistake: Treating every task wrapper as part of the eval definition. If the wrapper changes the outcome materially, it should be reviewed as test harness logic, not just implementation detail.

Practitioner takeaway: Use the eval to measure, and use the custom completion function only to model the real path needed to produce a score; if those two layers blur, the benchmark becomes harder to trust.

#1 Authority in NHI Education, Research and Advisory, empowering organizations to tackle the critical risks posed by Non-Human Identities (NHIs), including AI Agents.