What is the difference between vector search and eval-based search?

How the two search methods differ in practice

Vector search is best when the goal is semantic recall: you want passages, examples, or records that mean roughly the same thing, even if the wording differs. Eval-based search is best when the goal is behavioural recall: you want items that satisfy, violate, or resemble a rubric-defined condition, such as a control failure, unsafe tool use, or a repeated agent mistake.

That difference matters because the retrieval unit changes. Vector search clusters language by similarity in embedding space, so it is strong at synonyms, paraphrases, and related concepts. Eval-based search clusters by judged outcome, so it is strong at finding patterns that a human reviewer would mark the same way even when the surface text, wording, or workflow looks different.

In security workflows, that means vector search is useful for discovering “things that sound like this,” while eval-based search is useful for discovering “things that fail the same way.” The second approach is often more operationally useful when you are reviewing agent traces, policy violations, control exceptions, or incidents where the important signal is the outcome rather than the vocabulary.

Where vector search tends to win, and where it falls short

Vector search is usually the faster path when you have a natural-language question, a vague incident description, or a need to surface conceptually related material across a large corpus. It is also good when the search space contains messy prose, duplicated documentation, and near-duplicates that differ mainly in phrasing.

Its weakness is that proximity in semantic space is not the same as equivalence in policy or risk. Two items can be textually similar but operationally very different, especially when one item is safe, approved, or benign and the other is subtly noncompliant. In other words, vector search can over-retrieve “close enough” examples that do not actually meet the control condition you care about.

For that reason, teams often use vector search as a broad candidate generator, then apply a second-stage filter or reranker. That makes it a good front-end retrieval method, but not always the final decision method when precision against a control definition matters.

What eval-based search adds for security and agent review

Eval-based search is closer to a rubric-driven investigation tool than a semantic lookup tool. You define the criteria first, then retrieve examples that score similarly on those criteria, such as “tool used without authorization,” “policy bypass attempted,” or “failure to stop on an unsafe instruction.”

This is especially valuable when reviewing Breach and Attack Simulation (BAS) Guide-style control validation, because the interesting matches are often attack paths or failure patterns rather than matching wording. It is also useful when comparing agent behaviour against a rubric for Databricks Mosaic AI and Agent Bricks Security, where the issue may be whether the agent respected authorization, secrets handling, or logging expectations, not whether it used the same terms as the policy.

In practice, eval-based search can surface the “same failure, different text” cases that vector search misses. That makes it better for quality assurance, red teaming, and post-incident review where the objective is to find recurring control breakdowns, not just conceptually related content.

Practitioner Guidance

What to prioritise: Use vector search when you need breadth, discovery, or semantic recall; use eval-based search when you need precision against a known rubric, control, or failure pattern. If the question is “what is related to this?”, vector search fits; if the question is “what failed in the same way?”, eval-based search is the better tool.

What to verify: Check whether your retrieval objective is similarity of meaning or similarity of outcome. Mixing those two goals is the common mistake, because a semantically close result set can look good while still missing the control failures that matter operationally.

Practitioner takeaway: Treat vector search as the discovery layer and eval-based search as the judgement layer, especially when the real problem is identifying repeated bad behaviour rather than nearby language.

Related resources from NHI Mgmt Group

• Guide: Security Data Pipeline Guide
• What do security teams get wrong about search-based phishing?
• Why do RAG-based assistants create more risk than a normal search tool?
• What breaks when relationship-based access is flattened into vector metadata?
• How do organisations decide when search-based grading is worth the added cost?