Notifications
Clear all
Topic starter
19/08/2026 3:44 pm
TL;DR: Raw transcription accuracy is a near-tie across six speech-to-text providers and 240 audio cases, but meaning preservation diverges when models miss critical tokens like IDs, numbers, and action words, according to Braintrust’s evaluation. The practical lesson is that voice-agent teams need evals that measure downstream answer equivalence, not just word error rate, because small transcript errors can change the decision path.
NHIMG editorial — based on content published by Braintrust: Evaluating speech-to-text models
Questions worth separating out
Q: How should security teams evaluate voice agents that handle identity-sensitive requests?
A: They should test for task correctness, not just transcript similarity.
Q: Why do speech-to-text errors matter more in support and verification workflows?
A: Because those workflows depend on exact tokens that drive actions, such as IDs, serial numbers, invoice values, and request types.
Q: What do teams get wrong when they rely on word error rate alone?
A: They assume that a better-looking transcript automatically means a safer workflow.
Practitioner guidance
- Define critical-token classes for every voice workflow Catalogue the identifiers, amounts, action words, and verification phrases that must survive transcription in each workflow.
- Score downstream answer equivalence alongside transcript accuracy Run the same prompt against the reference transcript and the ASR output, then compare whether the answers are semantically the same.
- Attach reviewable audio to every evaluation trace Keep the original clip, transcript, and score in the same trace so reviewers can hear whether a low score reflects ASR failure, bad audio, or a flawed reference.
What's in the full article
Braintrust’s full blog covers the operational detail this post intentionally leaves for the source:
- Exact evaluation setup across six STT providers and 240 audio cases, including the scoring workflow
- Per-model latency and accuracy tables that show how the providers compared in practice
- Bucket-by-bucket breakdowns for customer support, finance, medical, ATC, and other domains
- Implementation notes on using traces, attachments, and concurrency controls to make the eval repeatable
👉 Read Braintrust’s evaluation of speech-to-text models for voice agents →
Speech-to-text accuracy is not the whole story for voice agents?
Explore further
19/08/2026 4:21 pm
Speech interfaces need identity-grade controls, not just ASR benchmarks. When a voice agent handles account support, order changes, or approvals, transcription becomes part of the identity and transaction control path. A nearly correct transcript can still redirect an action, which means evaluation must focus on whether the system preserved the decision-relevant token set. Practitioners should treat voice workflows as governed inputs, not convenience features.
A question worth separating out:
Q: How can organisations tell whether a voice model is good enough for production use?
A: They should look for three signals: high critical-token recall, stable answer equivalence, and a trace that lets reviewers hear the original audio. If a model performs well on ordinary speech but drops IDs or action verbs, it is not safe for workflows where exactness determines the outcome.
👉 Read our full editorial: Speech-to-text evals expose the real failure mode in voice agents
