Notifications
Clear all
Topic starter
19/08/2026 3:38 pm
TL;DR: Qwen3-8B completed every request through 256 concurrent users in TruFoundry’s vLLM benchmark, while Llama 3.1 8B delivered the highest raw throughput but hit 87% errors at 128 concurrent and Ministral 8B failed even at light load. For production AI gateways, throughput without error rates is a misleading planning metric.
NHIMG editorial — based on content published by TruFoundry: vLLM benchmark of Qwen3-8B vs Llama 3.1 8B vs Ministral 8B on a single A10
By the numbers:
- Ministral 8B showed 3.8% real HTTP errors at concurrency 4.
Questions worth separating out
Q: How should teams choose between a fast model and a stable model for AI gateways?
A: Choose the model that stays within acceptable latency and error bounds at the concurrency you actually expect, not the one with the best peak throughput.
Q: When does high throughput become a misleading metric for AI serving?
A: High throughput becomes misleading when it is achieved only after the system starts timing out or dropping requests.
Q: What should security and platform teams monitor in an AI gateway under load?
A: Monitor first-token latency, tail latency, error rate, queue depth, and backend admission behaviour.
Practitioner guidance
- Measure concurrency at production thresholds Run sweeps at the exact concurrency bands your gateway will face in production, including burst conditions above normal averages.
- Set workload-specific model policies Assign interactive chat, copilot, classification, and batch inference to different acceptance thresholds.
- Treat request shedding as a control failure When errors appear at low or moderate concurrency, investigate gateway queueing, memory pressure, and backend admission policy before scaling out.
What's in the full article
TruFoundry's full blog covers the benchmark mechanics this post intentionally leaves out:
- GuideLLM sweep configuration, including concurrency steps, cooldown periods, and workload definitions
- Full percentile tables across all tested concurrency levels for chat and classification
- The model-serving setup details for vLLM on a single A10 instance
- Gateway configuration and endpoint behaviour behind the benchmark results
👉 Read TruFoundry’s full vLLM benchmark of Qwen3, Llama, and Ministral →
vLLM under concurrency: which model is actually reliable?
Explore further
19/08/2026 4:18 pm
Peak throughput is not a governance metric. The article shows why a model that looks fast at high load can still be operationally unsuitable if it sheds requests or introduces long queue times. In AI gateway terms, the more important question is whether the serving layer preserves predictable execution for copilots, routers, and agent workflows. Practitioners should evaluate reliability as part of model governance, not as an afterthought.
A question worth separating out:
Q: How do you know if an AI model is suitable for interactive workloads?
A: An interactive model should hold a low first-token delay, stable p95 latency, and near-zero errors at the concurrency levels common in live use. If it only looks good in low-load testing, it is not yet proven for user-facing work. The safest signal is a flat latency curve under burst traffic rather than a high peak benchmark.
👉 Read our full editorial: vLLM benchmark shows Qwen3 holds under load while others fail
