vLLM benchmark shows Qwen3 holds under load while others fail

At a glance

What this is: This is a concurrency benchmark of three 8B models served through vLLM on a single NVIDIA A10, and the key finding is that Qwen3-8B stayed stable while the others degraded sharply under load.

Why it matters: It matters because AI gateway and platform teams need to size model serving around reliability, latency, and queue behaviour, not just peak token throughput, especially when AI agents or copilots depend on predictable runtime responses.

By the numbers:

• Qwen3-8B completed every request at every concurrency level through 256 with zero errors.
• Llama 3.1 8B hit 86.9% request failures at concurrency 128.
• Ministral 8B showed 3.8% real HTTP errors at concurrency 4.

👉 Read TruFoundry’s full vLLM benchmark of Qwen3, Llama, and Ministral

Context

Concurrency testing exposes the gap between a model that looks fast on paper and a model that stays usable under real traffic. In AI gateway operations, the critical question is not only how many tokens a model can emit, but whether it can sustain predictable service when multiple users, copilots, or AI agents are competing for the same runtime resources.

This benchmark is relevant to identity-adjacent AI governance because gateway reliability affects whether agentic systems can complete tasks, escalate cleanly, and preserve auditability. When a serving stack sheds requests or introduces long first-token delays, the operational failure can look like an application problem, but it often becomes a control problem for AI access, routing, and delegation.

For teams evaluating model-serving choices, the starting position in this article is typical: raw throughput is often celebrated before error behaviour and tail latency are examined. That ordering is backwards for production use.

Key questions

Q: How should teams choose between a fast model and a stable model for AI gateways?

A: Choose the model that stays within acceptable latency and error bounds at the concurrency you actually expect, not the one with the best peak throughput. For interactive workloads, predictable TTFT and low failure rates usually matter more than maximum tokens per second. For batch pipelines, throughput matters more, but only if retries and queue growth are already accounted for.

Q: When does high throughput become a misleading metric for AI serving?

A: High throughput becomes misleading when it is achieved only after the system starts timing out or dropping requests. If error rates rise sharply as concurrency increases, the peak number no longer reflects usable capacity. Teams should interpret throughput together with latency and failures, because raw capacity without reliability is not enough for production planning.

Q: What should security and platform teams monitor in an AI gateway under load?

A: Monitor first-token latency, tail latency, error rate, queue depth, and backend admission behaviour. Those signals show whether the gateway is preserving service quality or silently degrading into retries and request shedding. For agentic workflows, also track trace completeness, because failed requests can break downstream actions and obscure what actually happened.

Q: How do you know if an AI model is suitable for interactive workloads?

A: An interactive model should hold a low first-token delay, stable p95 latency, and near-zero errors at the concurrency levels common in live use. If it only looks good in low-load testing, it is not yet proven for user-facing work. The safest signal is a flat latency curve under burst traffic rather than a high peak benchmark.

Technical breakdown

Why TTFT and tail latency matter more than peak throughput

Time to first token, or TTFT, is the delay before a model produces its first output token. In interactive systems, that delay shapes user perception more than raw tokens per second because humans react to visible responsiveness, not batch efficiency. Tail latency matters because p95 captures the worst common experience, including queueing, scheduling, and decode contention. A model can have strong aggregate throughput while still feeling slow or unstable if prefills stack up or the gateway starts rejecting requests under burst load.

Practical implication: benchmark p50 and p95 latency at the concurrency levels your gateway actually serves, not just the highest token rate.

How vLLM scheduling and KV cache pressure shape stability

vLLM improves throughput by batching requests and managing decode steps efficiently, but its behaviour depends on request size, concurrency, and GPU memory headroom. The key constraint is KV cache pressure, because every active sequence occupies memory that the scheduler must preserve while admitting new requests. Once the queue grows too deep, prefill work competes with decode work and latency rises sharply. Architectural differences such as attention stabilisation and per-layer memory footprint can change how far a model stretches before the gateway starts shedding requests.

Practical implication: choose models that preserve scheduler headroom under your expected burst profile, not just the ones with the largest benchmark peak.

Why error rates are a control signal, not a side metric

Error rate tells you whether the serving stack is actually able to honour the workload you think it can support. In this benchmark, request shedding and 408 or 503 failures were not artefacts, they were the operational outcome of queue pressure and instability. For AI gateways, that becomes a governance issue because failed requests can break classification chains, tool calls, or agent workflows, and retries may amplify load. Reliable service means the model and gateway must survive both steady state and burst conditions.

Practical implication: treat sustained error rate above zero as a capacity defect until proven otherwise, especially for agentic or workflow-driven workloads.

NHI Mgmt Group analysis

Peak throughput is not a governance metric. The article shows why a model that looks fast at high load can still be operationally unsuitable if it sheds requests or introduces long queue times. In AI gateway terms, the more important question is whether the serving layer preserves predictable execution for copilots, routers, and agent workflows. Practitioners should evaluate reliability as part of model governance, not as an afterthought.

Latency stability is the real selection criterion for interactive AI. The benchmark demonstrates that first-token delay and p95 behaviour matter more than raw token output when users are waiting on a response. This is especially true for AI agents, where a stalled response can cascade into failed tool use, broken orchestration, or lost audit continuity. Teams should optimise for consistent service, not just high headline numbers.

Gateway control becomes more important when model behaviour diverges under load. When the same endpoint can serve multiple models with very different failure curves, the gateway is no longer just a routing layer. It becomes the place where performance policy, fallback logic, and service boundaries are enforced. That makes runtime governance part of availability engineering, not a separate concern.

Capacity planning for AI systems now includes behavioural variance. The article’s contrast between short chat and classifier workloads shows that one model can be reliable in one pattern and unstable in another. That is a reminder that AI governance cannot rely on a single benchmark number. Practitioners need workload-specific thresholds, because agentic systems and interactive copilots fail differently than batch pipelines.

Runtimes that govern access and traces need to stay deterministic under stress. For identity and access programmes, the important lesson is that AI gateway control planes must remain trustworthy when traffic spikes, because policy enforcement, traceability, and cost attribution all depend on stable request handling. Practitioners should treat degraded serving behaviour as a signal to re-check routing policy, fallback design, and operational boundaries.

What this signals

Latency governance is becoming a control-plane issue, not a model-choice issue. Once AI gateways sit between users, models, and agents, the practical question is whether the control plane can preserve policy, traceability, and bounded service when load spikes. For identity-led programmes, that means runtime reliability has to be evaluated alongside access rules, because an unstable gateway can undermine both auditability and delegation control.

Model selection now needs an operational boundary, not a preference ranking. The useful distinction is between workloads that can tolerate retries and those that cannot. That boundary should be reflected in service policies, fallback design, and incident playbooks, because AI systems that sit inside business workflows fail differently from isolated prototypes.

As agentic adoption grows, gateway stability becomes part of identity assurance. When an AI agent depends on a model endpoint for decisions, stalled responses and request failures can distort task completion and obscure accountability. Teams should align gateway monitoring with the same discipline they use for privileged access and service account control, because runtime behaviour is now part of the trust model.

For practitioners

• Measure concurrency at production thresholds Run sweeps at the exact concurrency bands your gateway will face in production, including burst conditions above normal averages. Record TTFT p50, TTFT p95, E2E p95, throughput, and error rate together so no single number hides instability.
• Set workload-specific model policies Assign interactive chat, copilot, classification, and batch inference to different acceptance thresholds. A model that is acceptable for async pipelines may still be poor for interactive use if first-token latency or request shedding crosses user tolerance.
• Treat request shedding as a control failure When errors appear at low or moderate concurrency, investigate gateway queueing, memory pressure, and backend admission policy before scaling out. If the system cannot sustain a clean operating point, do not normalise retries as a substitute for capacity.
• Separate routing governance from model selection Use the gateway to enforce model routing, fallback paths, and request traces independently from the serving backend. That separation makes it easier to compare runtime behaviour while preserving auditability across providers and models.

Key takeaways

• Throughput numbers are incomplete unless they are paired with tail latency and error behaviour.
• Qwen3-8B’s value in this benchmark is not raw speed but stable service under load, which matters more for interactive AI.
• AI gateway teams should set concurrency policy by workload, because batch-friendly models can still fail interactive use cases.

Key terms

• Time To First Token: Time to first token is the delay between a request reaching the model and the first output token being returned. It is one of the clearest measures of perceived responsiveness in interactive AI systems, because long first-token waits make an application feel slow even when total throughput is high.
• Tail Latency: Tail latency is the slow end of a system's performance distribution, usually measured with metrics such as P95 or P99. Security teams care about it because the outliers are often the scans, checks, or reviews that block production workflows and erode trust in the control.
• Request Shedding: Request shedding is the practice, intentional or not, of dropping or rejecting incoming work when a system is overloaded. In AI gateways and model-serving stacks, it is a sign that the runtime has exceeded its practical concurrency limit, and it can undermine workflow completion and audit continuity.
• KV Cache: A KV cache stores key and value tensors produced during transformer attention so later tokens can reuse them instead of recomputing the entire prompt. In production serving, cache locality strongly affects latency, throughput, and GPU cost because the cache lives on the node that processed the earlier tokens.

What's in the full article

TruFoundry's full blog covers the benchmark mechanics this post intentionally leaves out:

• GuideLLM sweep configuration, including concurrency steps, cooldown periods, and workload definitions
• Full percentile tables across all tested concurrency levels for chat and classification
• The model-serving setup details for vLLM on a single A10 instance
• Gateway configuration and endpoint behaviour behind the benchmark results

👉 The full TruFoundry post includes the percentile tables, sweep setup, and serving configuration behind these results.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It gives security and identity practitioners a structured way to connect runtime control with access governance.