Custom facets make AI agent trace review operationally useful

At a glance

What this is: This is a how-to guide for designing custom facets in AI agent trace review, with the key finding that recurring review questions need stable, narrowly scoped labels when built-in facets are too broad.

Why it matters: It matters because AI governance teams need trace labels they can trust for quality review, failure monitoring, and roadmap decisions across agentic AI and broader IAM-adjacent programmes.

👉 Read Braintrust's guide to designing custom facets for AI agent traces

Context

AI agent observability becomes useful only when teams can classify traces in a way that supports repeatable review, not just ad hoc inspection. Built-in facets such as Task, Sentiment, and Issues are broad by design, but production teams often need labels that reflect specific outcomes, failure modes, or business decisions. For identity and governance teams watching agentic AI adoption, that same problem shows up as an inability to distinguish routine automation from risky behaviour, which is why precise classification matters.

Braintrust’s approach is about turning recurring questions into a controlled taxonomy. The operational challenge is familiar to security and IAM programmes: if the label set is too vague, output becomes noisy; if the slice is too wide, the classifier loses the signal. This is especially relevant when AI agents participate in workflows that touch sensitive data, tool use, or access decisions, where review needs to be stable enough for audit and workflow routing.

Key questions

Q: How should teams design custom facets for AI agent traces?

A: Start with one operational question, define mutually exclusive labels, and make sure each label routes to a concrete workflow. Then choose the narrowest trace slice that contains the evidence needed for that label. If the facet cannot produce stable results across reruns, it is not ready for production review.

Q: Why do broad trace labels fail in AI agent governance?

A: Broad labels collapse too many behaviours into the same bucket, so the classifier cannot separate meaningful patterns from noise. That creates unstable clustering, weak trend analysis, and low-confidence review output. Governance teams need labels that are narrow enough to stay consistent and specific enough to support action.

Q: What do security teams get wrong about observability labels for AI agents?

A: They often treat labels as reporting categories instead of decision tools. If a label does not drive review, escalation, filtering, or scoring, it does not improve governance. The better approach is to design the facet around the action the team actually intends to take after classification.

Q: How do you know if a custom facet is actually working?

A: A facet is working when the same trace gets the same label on rerun, the labels are easy to interpret, and each output clearly maps to a downstream workflow. If clusters are noisy or labels overlap, the taxonomy or preprocessor needs to change before the facet can be trusted.

Technical breakdown

When built-in trace facets stop being useful

Built-in facets are designed to capture broad patterns across traces, not the detailed questions product teams ask every week. Task, Sentiment, and Issues are useful first-pass dimensions, but they collapse too much variation when a team needs to separate recurring operational slices such as refund handling, tool failures, or research quality. A custom facet solves that by defining one question, one label space, and one downstream workflow. The important design principle is consistency: the same trace should receive the same label across reruns if the input slice and taxonomy stay stable.

Practical implication: create a custom facet when built-in labels force repeated manual filtering for the same review question.

How preprocessors shape the evidence a facet can see

A preprocessor determines which parts of a trace reach the classifier. That matters because facet quality is limited by the evidence you expose. User-message-only preprocessors work well when the label depends on request intent or customer tone, while errors-and-context preprocessors work better when the label depends on failure windows, retries, or tool exceptions. Metadata projection is useful when classification depends on environment, plan tier, or model version. The common rule is to pass the narrowest trace slice that still contains the signal needed for the label.

Practical implication: tune the preprocessor before tuning the prompt, because missing or noisy input cannot be fixed downstream.

Why stable labels and exclusion rules matter

A custom facet only works if its labels are mutually exclusive, stable across reruns, and tied to a concrete workflow. Overlapping labels produce inconsistent clustering because similar traces can land in different buckets on different runs. Exclusion regex is the final safeguard, removing empty outputs such as NONE or N/A so they do not create noisy clusters. The point is not just classification accuracy but operational usefulness: each label should route a trace to review, scoring, escalation, or filtering. Without that link, the facet adds complexity without improving decisions.

Practical implication: define labels so each one maps to a distinct review or action path, then exclude empty classifications from clustering.

NHI Mgmt Group analysis

Custom facets are really governance controls for AI agent observability. The article shows that classification quality depends on whether the team can define a narrow, repeatable question before it tries to automate review. That is a governance pattern, not just a product feature, because the label taxonomy becomes the control surface for dashboards, evals, and alerts. In agentic AI environments, poor trace classification creates blind spots similar to weak identity telemetry. Practitioners should treat facet design as part of operational governance, not a reporting convenience.

The named concept here is facet drift. When labels are broad, overlapping, or tied to noisy input, the same trace can move between categories across runs. That instability undermines trust in the classification layer and makes trend analysis unreliable. For teams governing AI agents, facet drift is the observability equivalent of inconsistent policy enforcement: if the signal changes from run to run, the control cannot support decisions. Practitioners should require stable label boundaries before using facet outputs in reviews or alerts.

Braintrust’s guidance also mirrors a core identity lesson: the narrowest useful context usually produces the best control outcome. That is true in IAM, and it is true in AI trace governance. User-only slices, failure-window slices, and metadata projection each reduce noise by focusing the classifier on the evidence that matters. The broader lesson for AI governance teams is that observability should be designed around decision quality, not data volume. Practitioners should align trace slicing with the decision the facet is meant to support.

AI agent trace review becomes more valuable when the taxonomy matches a real workflow. Labels such as GRANTED, ESCALATED, or FAILURE only matter if they route work somewhere meaningful. That is the same logic identity teams use when they tie entitlements to approval, review, or revocation paths. In practice, a facet that cannot drive a follow-up action is only decoration. Practitioners should build facets around the review process they actually intend to run.

For agentic AI programmes, the bigger risk is not lack of data, but lack of classification discipline. Teams often have traces, logs, and model outputs already, yet still cannot answer basic operational questions because their categories are too vague. That makes custom facets a control design exercise, not a taxonomy exercise. Practitioners should use them to turn recurring trace patterns into durable governance signals, especially where AI agents touch customer data, tools, or escalation paths.

What this signals

Facet design is becoming a practical governance layer for agentic AI programmes, especially where teams need reliable review signals rather than broad telemetry. The strongest pattern is to treat trace classification like policy design: narrow the input, narrow the label set, and align the output to a specific action. Facet drift: when labels move between categories across reruns, the observability layer stops being decision-grade and starts becoming noise.

For identity and AI governance teams, the real question is whether trace labels can support audit, escalation, and remediation without manual reinterpretation. That is why stable preprocessing matters as much as prompt design. The control objective is not perfect classification, but classification that is consistent enough to support accountable operation across the agent lifecycle.

For practitioners

• Define one review question per facet Start with a single operational question such as churn risk, tool reliability, or citation quality, then build labels only for that decision. If the facet tries to answer several questions at once, stability and clustering quality will suffer.
• Use the narrowest trace slice that contains the signal Choose user-only, error-window, or metadata-projected input based on the label you need. Narrow slices reduce noise and make reruns more consistent, especially when assistant output or tool scaffolding obscures the evidence you actually care about.
• Tie every label to a downstream workflow Make sure each label routes to a concrete next step, such as review, escalation, filtering, or scoring. Labels without a follow-up action add volume to Topics but do not improve governance or product decisions.
• Exclude empty outputs from clustering Use an exclusion regex for outputs such as NONE or N/A so empty classifications do not become noisy clusters. Review the excluded traces after testing to confirm the regex is not removing valid edge cases.

Key takeaways

• Custom facets turn AI agent trace review into a decision workflow when built-in labels are too broad.
• Facet quality depends on narrow preprocessing, stable labels, and a clear downstream use for every category.
• Teams should treat trace classification as a governance control, because inconsistent labels undermine auditability and actionability.

Key terms

• Custom Facet: A custom facet is a label system built to classify traces by a specific operational question rather than a generic category. It combines a preprocessor, a prompt, and optional exclusion rules so teams can turn recurring review needs into consistent, workflow-linked outputs.
• Preprocessor: A preprocessor is the step that selects which parts of a trace are shown to the classifier. It can pass user messages, error windows, or metadata fields, and its job is to narrow the evidence so the label is based on the signal that matters, not irrelevant trace noise.
• Facet Drift: Facet drift is the instability that appears when the same trace receives different labels across reruns. It usually comes from vague label boundaries, noisy inputs, or inconsistent preprocessing, and it undermines the reliability of dashboards, review workflows, and trend analysis.
• Exclusion Regex: An exclusion regex is a pattern that removes unwanted facet outputs from clustering, most often empty classifications such as NONE or N/A. It prevents low-value results from becoming clusters and helps keep Topics focused on labels that represent real operational decisions.

What's in the full article

Braintrust's full article covers the implementation detail this post intentionally leaves at the governance level:

• Preprocessor patterns for user-only traces, failure windows, and metadata projection in production facets
• Prompt structures and label boundaries that improve consistency across repeated reruns
• Exclusion regex handling for NONE and N/A outputs so empty classifications do not pollute clustering
• Worked examples for support, coding, multilingual chatbots, and sales or PLG agents

👉 The full Braintrust article covers preprocessor design, label stability, and facet tuning examples across several agent types.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, workload identity, and agentic AI identity. It helps security practitioners connect identity controls to the operational realities of modern automation and access governance.