Custom facets make AI agent trace review operationally useful

At a glance

What this is: Braintrust argues that custom facets are needed when built-in AI agent trace labels are too coarse to support recurring review questions, workflow routing, and product decisions.

Why it matters: This matters because IAM, NHI, and AI governance teams increasingly need stable classifications that connect observed agent behaviour to review, evaluation, and operational controls.

Context

AI agent trace review becomes unreliable when labels are too broad to separate the operational questions teams actually need to answer. In practice, generic buckets such as task type, sentiment, or issues can blur materially different cases together, which weakens evaluation, routing, and product decision-making.

Custom facets address that governance gap by turning repeated review logic into a stable classification layer. Where an AI system behaves like a software entity acting with delegated tools and context, the control problem shifts from merely observing traces to consistently classifying the patterns that matter for oversight, quality, and escalation.

Key questions

Q: What should teams do when built-in AI agent trace labels are too broad?

A: Create a custom facet only when the team keeps asking the same operational question and the built-in labels cannot answer it at the right level of detail. Start with one decision, one label space, and one downstream workflow. If the labels do not change what the team does next, the facet is probably too vague to be useful.

Q: Why do broad trace labels fail in AI agent governance?

A: Broad labels collapse too many behaviours into the same bucket, so the classifier cannot separate meaningful patterns from noise. That creates unstable clustering, weak trend analysis, and low-confidence review output. Governance teams need labels that are narrow enough to stay consistent and specific enough to support action.

Q: What are the signs that a custom facet taxonomy is failing?

A: The most common signs are heavy use of catch-all labels, overlapping categories, inconsistent results on rerun, and clusters that do not map cleanly to any workflow. If reviewers still need manual SQL filters after launch, the facet is not precise enough to replace the recurring review slice.

Q: How should AI teams decide whether a custom facet belongs in production?

A: Treat production readiness as a workflow test, not a taxonomy test. A facet belongs in production when it classifies a narrow slice consistently, feeds a concrete action such as routing or scoring, and still behaves the same after a rerun on historical traces. If any of those conditions fail, it is still experimental.

Technical breakdown

Why built-in trace facets break down in production AI agents

Built-in facets such as Task, Sentiment, and Issues are useful when teams need coarse triage, but they break down when the review question is domain-specific. If a support agent, coding assistant, and sales agent all land in the same broad bucket, the label stops being operationally meaningful. The underlying problem is taxonomy granularity: the classifier can only separate what the label set makes explicit. In AI governance terms, this is a measurement problem, not just a UI problem, because weak label boundaries create unstable clusters and untrustworthy trend lines.

Practical implication: define trace dimensions around the decision you need to make, not around the broad category the model can guess.

How custom facets turn recurring filters into stable governance signals

A custom facet is a repeatable classification layer built from three pieces: a preprocessor, a prompt, and an exclusion rule. The preprocessor decides which evidence the model sees, the prompt defines valid labels and output shape, and the exclusion regex prevents empty or irrelevant outputs from polluting clustering. This matters because stability comes from controlling the input slice as much as the taxonomy. For agent operations, the same recurring SQL filter or manual review slice can become a facet that feeds dashboards, scoring, and workflow routing consistently over time.

Practical implication: standardise the trace slice first, then turn the recurring review logic into a facet with clear downstream use.

Why label design determines whether trace review is trustworthy

Facet quality fails most often at the label-design stage. Good labels are mutually exclusive, narrow, and tied to a concrete workflow, such as routing denied refunds to policy review or failed tool calls to reliability analysis. Broad labels like quality or customer service request create overlap, which increases classifier variance across reruns and makes the output hard to trust. In agent governance, that means the taxonomy itself becomes part of the control surface, because inconsistent labels undermine both evaluation and accountability.

Practical implication: use one label space for one decision and require each label to point to a specific next step.

NHI Mgmt Group analysis

Custom facets are a governance control, not just an analytics convenience. When AI agent traces become operational evidence for quality, routing, or escalation, the classification layer is part of the control environment. Broad labels create false confidence because they look measurable while hiding the distinctions practitioners actually need. The discipline here is to design the taxonomy around decision points, not around whatever the model can label loosely.

AI agent observability now depends on classification stability. If the same trace can land in different buckets across reruns, the resulting dashboards are not operationally trustworthy. That is especially important where AI agents act through tools, because the governance question is not only what happened, but whether the same pattern can be recognized again under the same rules. The stable-facet concept is becoming a basic observability requirement.

Trace taxonomies will increasingly define how AI programmes scale oversight. As teams add support, coding, research, and sales agents, the default built-in facets will not keep pace with programme-specific risk and quality questions. The article shows that the next governance layer is not more traces, but more precise labels that map to review workflows, evals, and product decisions. Practitioners should treat facet design as a reusable control pattern, not a one-off experiment.

Agent review needs a named concept: classification-to-workflow fit. A facet is only useful when each label connects directly to a review, alert, or release decision. That fit is what separates a cosmetic taxonomy from an operational control. For practitioners, the key test is whether the label set changes what the team does next.

What this signals

Classification-to-workflow fit is becoming a core AI governance pattern. The lesson for practitioners is that observability only becomes operational when labels map to actions. That pushes AI programmes toward narrower taxonomies, cleaner trace slices, and review loops that are designed around business decisions rather than generic monitoring.

Agent trace review will increasingly separate surface behaviour from actionable evidence. As teams move beyond simple sentiment or issue tagging, the useful question becomes which trace dimension predicts escalation, routing, or quality failure. That shift matters for AI agents specifically, because delegated tool use creates richer failure patterns than a standard chat transcript.

Facet design should now be treated as part of model operations governance. Teams that rely on agent traces for releases, support triage, or product feedback need a repeatable way to classify evidence before they can trust dashboards or evals. The operational gain comes from stable labels, not from more labels.

For practitioners

• Design facets around one decision point Anchor each custom facet to a single operational question such as refund disposition, tool reliability, or churn risk, and avoid label sets that mix unrelated dimensions like tone and outcome.
• Narrow the preprocessor to the evidence slice Pass only the trace spans needed for the facet, such as user messages, error windows, or selected metadata, so the classifier sees consistent evidence on every run.
• Use mutually exclusive labels with explicit workflow routing Give each label one downstream action, such as review, escalation, filtering, or scoring, so the taxonomy supports a concrete operating process instead of general tagging.
• Re-test after launch and after taxonomy changes Re-run the facet on historical traces, inspect the cluster distribution, and adjust labels or exclusion rules when reruns show instability or noisy low-value clusters.

Key takeaways

• Built-in trace facets are useful for broad triage, but they are often too coarse to support the recurring review questions production AI agents generate.
• Custom facets work best when they classify one decision, use mutually exclusive labels, and connect directly to a workflow or escalation path.
• Facet stability matters because rerun variance turns trace review into an observation exercise instead of a reliable governance control.

Key terms

• Custom Facet: A domain-specific label extracted from traces to classify behaviour along a dimension built around the product or control question. It extends generic task or sentiment analysis by making logs useful for governance, compliance, or specialised operational review.
• Preprocessor: A preprocessor is the step that selects which parts of a trace are shown to the classifier. It can pass user messages, error windows, or metadata fields, and its job is to narrow the evidence so the label is based on the signal that matters, not irrelevant trace noise.
• Exclusion Regex: An exclusion regex is a pattern that removes unwanted facet outputs from clustering, most often empty classifications such as NONE or N/A. It prevents low-value results from becoming clusters and helps keep Topics focused on labels that represent real operational decisions.
• Label Stability: Label stability means the same trace receives the same classification when the facet is run again. Stability depends on consistent input, clear label boundaries, and a prompt that limits classifier ambiguity, making the output dependable enough for operational use.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and machine identity security. It is designed for practitioners who need a stronger control model for delegated systems and the identities they use.