Topic starter
19/08/2026 3:38 pm
TL;DR: Qwen3-8B completed every request through 256 concurrent users in TruFoundry’s vLLM benchmark, while Llama 3.1 8B delivered the highest raw throughput but hit 87% errors at 128 concurrent and Ministral 8B failed even at light load. For production AI gateways, throughput without error rates is a misleading planning metric.
Editorial analysis by NHI Mgmt Group, based on content published by TruFoundry: “vLLM Benchmark: Qwen3-8B vs Llama 3.1 8B vs Ministral 8B on a single A10”.
Key questions
Q: What breaks when a model is sized only by peak throughput?
A: Peak throughput can hide the point where queueing, timeouts, and request shedding begin.
Q: Why do some AI models look fast but still feel slow to users?
A: Users experience the delay before the first token far more than the backend’s aggregate token rate.
Q: What are the signs that an AI gateway is at its concurrency limit?
A: The main signs are rising error rates, timeout responses, widening p95 latency, and a gap between nominal throughput and completed requests.
Practitioner guidance
- Define separate concurrency policies for interactive and batch traffic Set a stricter limit for chat and copilot endpoints, then allow higher concurrency only for async or retry-tolerant jobs where a late response is still useful.
- Benchmark completion rate at each load step Run sweeps across the intended concurrency bands and record error rate, TTFT p50 and p95, and end-to-end latency for every candidate model on the actual GPU class.
- Treat gateway routing as a policy layer Use the gateway to separate user-facing traffic from background workloads so routing, fallback, and rate limits reflect the service class rather than the model alone.
Bottom line: Throughput numbers alone do not describe whether a model can sustain real traffic without timeouts or dropped requests.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
19/08/2026 4:18 pm
Throughput-only planning is now a governance error. Capacity decisions for AI gateways cannot be made from peak tokens per second alone, because the useful unit is completed requests at an acceptable latency profile. This benchmark shows that a model can look stronger on paper while shedding traffic under realistic load, which is the wrong metric for production governance. Practitioners should treat completion reliability as the primary control objective, not an afterthought.
A few things that frame the scale:
- AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
A question worth separating out:
Q: How should teams decide between interactive and batch model serving?
A: Use interactive serving when users are waiting on immediate output and reliability is more important than peak token rate. Use batch serving when retries are acceptable and higher raw throughput justifies looser latency. The decision should follow the workload, not the model brand or the best benchmark headline.
👉 Read our full editorial: vLLM benchmark shows Qwen3 holds under load while others fail