Notifications
Clear all
Topic starter
18/08/2026 2:21 pm
TL;DR: Built-in Kimi K3 and DeepSeek V4 Flash models can now be evaluated inside Braintrust across playgrounds, prompts, scorers, and the gateway without separate inference setup, with the comparison framed around quality, latency, and token usage. The governance issue is not model access alone, but whether evaluation workflows preserve reproducibility, traceability, and prompt discipline before teams promote a model into production.
NHIMG editorial — based on content published by Braintrust: Compare Kimi K3 and DeepSeek V4
By the numbers:
- The article benchmarks 327 dialogs from MathTutorBench to compare model responses, response time, and token usage in one run.
- DeepSeek V4 Flash produced 365 median completion tokens in the comparison, including hidden reasoning tokens.
- GLM-5.2 recorded a 0.94-second first visible token time with reasoning disabled in the probe.
Questions worth separating out
Q: How should security teams evaluate GenAI models before production?
A: Security teams should test models with realistic adversarial scenarios, including direct prompt attacks and indirect instruction injection through retrieved content.
Q: Why do AI platform errors create identity risk for IAM teams?
A: AI platform errors can expose tokens, internal identifiers, or backend state because many orchestration systems return rich diagnostic payloads.
Q: What breaks when prompt sets and scorers are not versioned?
A: The benchmark loses comparability.
Practitioner guidance
- Separate evaluation access from deployment access Grant different roles for model testing, scorer management, and production gateway use.
- Version prompts, datasets, and scorers as governed assets Treat benchmark prompts and scoring logic as controlled records with change history, review ownership, and rollback.
- Scope AI gateway credentials to specific teams and workloads Use short-lived credentials and per-workload service identities for evaluation traffic.
What's in the full article
Braintrust's full blog covers the operational detail this post intentionally leaves for the source:
- Side-by-side scoring methodology for the three models on the 327-dialog MathTutorBench run
- Step-by-step guidance for running prompts, scorers, and production traces through the gateway
- The exact SDK configuration used to point OpenAI and Anthropic clients at the Braintrust gateway
- Token and latency observations from high-reasoning versus disabled-reasoning probes
👉 Read Braintrust's comparison of Kimi K3 and DeepSeek V4 Flash in built-in evaluations →
Built-in open models: what it means for AI evaluation controls?
Explore further
18/08/2026 3:13 pm
Built-in evaluation platforms are becoming governance systems, not just testing tools. Once model comparison, scoring, and deployment happen in one workflow, identity controls become part of the evaluation process itself. Access to prompts, datasets, and model-switching functions now shapes the quality of the decision, not just the efficiency of the workflow. Practitioners should treat model evaluation permissions as a governance boundary, not an admin convenience.
A question worth separating out:
Q: Who is accountable when an AI model is promoted from a controlled evaluation into production?
A: Accountability should sit with the owners of the evaluation workflow, the model risk process, and the identity controls that governed access to the test environment. If the decision path is not auditable, no one can prove the promotion was made on repeatable evidence.
👉 Read our full editorial: Built-in open models change how teams evaluate AI agents
