vLLM benchmark shows Qwen3 holds under load while others fail

At a glance

What this is: This is a vLLM concurrency benchmark showing that one model can sustain every request under load while another posts better raw throughput but fails at higher concurrency.

Why it matters: It matters because AI gateway and platform teams need capacity plans based on completed requests, not just tokens per second, when deciding what can safely serve interactive or batch workloads.

Context

A concurrency benchmark matters when teams are choosing models for shared AI gateways, because raw throughput alone does not tell you whether requests are actually completing under real load. The operational question is whether latency, queueing, and error rates remain acceptable as simultaneous users rise.

This article is about serving behaviour in an AI gateway, not model quality in the abstract. That puts the control problem on runtime reliability, request shedding, and traffic shaping, with a clear identity-adjacent angle where the gateway mediates access to model endpoints and usage policy.

Key questions

Q: What breaks when a model is sized only by peak throughput?

A: Peak throughput can hide the point where queueing, timeouts, and request shedding begin. A model may still produce many tokens per second while failing a large share of requests at higher concurrency, which makes it unsafe for interactive traffic. Teams should size for completed requests, not just tokens delivered under ideal conditions.

Q: Why do some AI models look fast but still feel slow to users?

A: Users experience the delay before the first token far more than the backend’s aggregate token rate. A model with strong decode speed can still feel sluggish if its time to first token rises sharply as concurrency increases. For chat and copilot use cases, the first token is the service quality signal that matters most.

Q: What are the signs that an AI gateway is at its concurrency limit?

A: The main signs are rising error rates, timeout responses, widening p95 latency, and a gap between nominal throughput and completed requests. When the gateway starts shedding traffic or the first-token delay climbs faster than request volume, the model has moved beyond its stable operating envelope.

Q: How should teams decide between interactive and batch model serving?

A: Use interactive serving when users are waiting on immediate output and reliability is more important than peak token rate. Use batch serving when retries are acceptable and higher raw throughput justifies looser latency. The decision should follow the workload, not the model brand or the best benchmark headline.

Technical breakdown

Why concurrency exposes gateway instability

Concurrency testing holds a fixed number of requests in flight and measures what happens as that load rises. In serving stacks like vLLM, the failure point is often not model accuracy but scheduler pressure, KV-cache growth, and request queue saturation. A model can look fast in isolation and still fail once prefills, decodes, and streaming responses compete for the same GPU and gateway budget. That is why concurrency sweeps are more useful than single-point latency tests for production planning.

Practical implication: capacity planning has to include completion rate and tail latency at each concurrency step, not just best-case throughput.

Why first-token latency matters more than peak throughput for chat

Time to first token, or TTFT, is the user-visible delay before a response starts streaming. For copilot and chat workloads, TTFT is usually more important than raw token throughput because users feel the pause before the first word, not the backend’s aggregate token rate. A model that decodes quickly after a slow start can still feel worse than a model with steadier initial response times. Gateway design therefore needs to distinguish interactive and batch workloads instead of treating all tokens as equal.

Practical implication: set interactive service targets around TTFT and error-free completion, then separate batch traffic into a different concurrency policy.

How model architecture changes load behaviour in vLLM

Different transformer architectures consume GPU memory and scheduler time differently, even when parameter counts are similar. Smaller per-layer cache footprints and attention stabilisation can leave more room for queued requests, while larger feed-forward layers can boost token output once a request is underway but reduce headroom for new prefills. In practice, that means two 8B models can behave very differently under the same vLLM configuration. The serving layer must be benchmarked as a system, not assumed from model size alone.

Practical implication: benchmark candidate models on the exact gateway and GPU combination you intend to run before committing traffic policies.

NHI Mgmt Group analysis

Throughput-only planning is now a governance error. Capacity decisions for AI gateways cannot be made from peak tokens per second alone, because the useful unit is completed requests at an acceptable latency profile. This benchmark shows that a model can look stronger on paper while shedding traffic under realistic load, which is the wrong metric for production governance. Practitioners should treat completion reliability as the primary control objective, not an afterthought.

Interactive and batch traffic need different control assumptions. The benchmark separates a chat-like workload from a classifier-like workload, and the results diverge in ways that matter for operating policy. That is the clearest sign that one concurrency limit cannot govern every model use case. Gateway policy should reflect user experience, retry tolerance, and queue behaviour rather than one generic throughput ceiling.

Model serving is an access-control problem as much as a performance problem. Once an AI gateway becomes the enforcement point for who or what can reach a model, it carries identity and policy implications even when the article is about latency. Request routing, rate limits, and fallback behaviour are effectively governance controls for machine clients. The practitioner conclusion is to evaluate the gateway as part of AI access governance, not just infrastructure throughput.

Load testing should surface the failure cliff before users do. The important operational question is where the first unrecoverable errors begin, not where the highest benchmark number appears. That requires testing at the same concurrency bands and response modes the production gateway will actually see. Teams should use the benchmark pattern here as a template for proving safe operating envelopes before production rollout.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: Enterprise AI Copilot Security Guide

What this signals

Capacity governance is shifting from model selection to service-class design. The practical question is no longer which model can output the most tokens in a clean run, but which traffic class it can serve without degrading the user experience. That means concurrency ceilings, fallback routing, and retry policy now belong in the same decision set as model choice.

AI gateway teams need a named operating concept for this problem: concurrency reliability. It is the point at which a model remains functionally usable as simultaneous demand rises, and it is defined by completed requests, not headline throughput. Teams that do not measure that boundary will misread load tests and overstate production readiness.

For practitioners

• Define separate concurrency policies for interactive and batch traffic Set a stricter limit for chat and copilot endpoints, then allow higher concurrency only for async or retry-tolerant jobs where a late response is still useful.
• Benchmark completion rate at each load step Run sweeps across the intended concurrency bands and record error rate, TTFT p50 and p95, and end-to-end latency for every candidate model on the actual GPU class.
• Treat gateway routing as a policy layer Use the gateway to separate user-facing traffic from background workloads so routing, fallback, and rate limits reflect the service class rather than the model alone.
• Validate model choice on the exact serving stack Repeat the benchmark on the same vLLM version, endpoint style, and GPU tier you plan to run in production before using throughput numbers for capacity planning.

Key takeaways

• Throughput numbers alone do not describe whether a model can sustain real traffic without timeouts or dropped requests.
• The benchmark’s useful signal is the gap between raw token output and completed requests as concurrency increases.
• Production AI gateways should be tuned to workload class, with separate limits for interactive chat and retry-tolerant batch jobs.

Key terms

• Concurrency Sweep: A concurrency sweep is a benchmark method that holds a fixed number of requests in flight while increasing load in steps. It reveals when latency, queueing, and error rates begin to degrade under real traffic pressure, which is more useful than a single isolated throughput result.
• Time-to-First-Token: Time-to-First-Token is the initial delay before an AI model emits its first response token. It is a user-experience and control signal because it reflects how quickly the system begins to respond, and it often reveals latency, routing, or workload issues before broader service failures appear.
• AI Gateway: A control point that sits between AI applications and the models, tools, or data they call. In practice, it can authenticate requests, enforce policy, inspect runtime behaviour, and stop unsafe actions before they spread into connected systems.
• Error Rate Under Load: Error rate under load is the proportion of requests that fail as concurrency rises. It is a critical production signal because a system with strong headline throughput can still be unusable if too many requests time out or return errors when demand spikes.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and machine identity security. It helps practitioners connect identity control to the broader security and AI operations their programmes depend on.