Mixture of Experts (MoE) Explained: Total vs Active

Mixture of experts (MoE) is a neural network design in which each layer holds many smaller sub-networks, called experts, and a learned router sends each token to only a few of them, so a model can have a very large total parameter count while using only a fraction of it for any one token. Many open-weight large language models published in 2026 describe themselves this way, quoting two numbers such as "2.4T total, 95B active". This page explains what those numbers mean, how routing works, where the idea came from, what it changes for cost, memory and hardware, and what each provider states about its own models. It is written for learners first and then for the engineers, architects and buyers who have to size, host or compare these models.

Last checked: 2026-11-15. Providers are listed alphabetically; order is never a ranking.

This is an independent reference page from NHI Mgmt Group based on public provider documentation; it is not sponsored or endorsed by any provider.

Key takeaways

Two parameter counts, two meanings: total parameters decide how much memory the weights need; active parameters decide roughly how much compute each token costs.
A router picks the experts: a small gating network scores the experts for every token and only the top few (for example 2, 8, 10 or 16) run, with some designs adding always-on shared experts.
The idea is older than today's chatbots: the sparsely-gated MoE layer was published in January 2017, and Switch Transformers took it to trillion-parameter scale in 2021.
Open-weight cards publish the numbers, closed APIs mostly do not: Alibaba Qwen, Mistral AI, Moonshot AI, NVIDIA, OpenAI (gpt-oss), Tencent and others state total and active parameters; Anthropic, Google, OpenAI (GPT-6 series) and xAI do not state architecture on their model pages.
Active is not the same as cheap to host: every expert must sit in memory, so a model with few active parameters can still need many GPUs.

What is mixture of experts in plain language?

A conventional ("dense") transformer pushes every token through every parameter in every layer. In an MoE transformer, the feed-forward block of a layer is replaced by a set of experts. A router looks at the token's internal representation, scores each expert, and passes the token to the highest-scoring few. Their outputs are combined, weighted by the router scores, and the token moves on to the next layer. A different token, or the same token in a different layer, may go to different experts.

An everyday analogy: a large hospital employs hundreds of specialists, but each patient sees only two or three of them. The hospital's total expertise is large; the cost of one visit is small. The analogy has a limit: MoE experts do not specialise in human-readable topics such as "law" or "medicine" by design. What each expert learns emerges from training, and providers' model cards do not claim topic-level specialisation.

Key terms

Term	Meaning
Expert	One feed-forward sub-network inside an MoE layer.
Router or gating network	A small trainable network that scores experts for each token and selects which ones run.
Top-k routing	Selecting the k highest-scoring experts per token per layer (top-2 in Mixtral 8x7B, top-8 in Tencent Hy3).
Shared expert	An expert that processes every token in addition to the routed ones (Qwen3.8 activates 1 shared expert; Kimi K3 lists 2).
Total parameters	Every weight in the model, including all experts. Drives storage and memory size.
Active (activated) parameters	The weights actually used to process one token. Drives compute per token.
Sparse vs dense	Sparse models activate a subset of weights per token; dense models activate all of them.
Naming such as A95B or 30B-A3B	A convention in open-weight model names: total size first, then "A" and the active size (Qwen3.8-2.4T-A95B, Nemotron 3.5 Lightning 30B-A3B).

Where did mixture of experts come from?

The table below lists the milestones confirmed from the original papers and model cards. Dates are submission or release dates as the source states them.

Date	Milestone	What the source states
January 2017	"Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" (Shazeer, Mirhoseini, Maziarz, Davis, Le, Hinton, Dean)	A layer of up to thousands of feed-forward experts with a trainable gating network that picks a sparse combination per example; reports more than 1000x improvements in model capacity with minor efficiency losses, and MoE models of up to 137 billion parameters between stacked LSTM layers.
January 2021 (v3 June 2022)	"Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity" (Fedus, Zoph, Shazeer)	A simplified routing algorithm to cut communication and compute cost; up to 7x faster pre-training than T5-Base and T5-Large on the same compute; a trillion-parameter model with a 4x speedup over T5-XXL; names training instability as a barrier and proposes techniques that also allow bfloat16 training.
January 2024	"Mixtral of Experts" (Mistral AI)	Eight feed-forward experts per layer, a router selecting two per token, about 47B total and 13B active parameters, Apache 2.0 licence.
2025 to 2026	Open-weight frontier releases	Model cards from Alibaba Qwen, DeepSeek, MiniMax, Mistral AI, Moonshot AI, NVIDIA, OpenAI, Tencent and others publish MoE models from 30B to 2.8T total parameters (see the table below).

How does routing work, step by step?

Token arrives at an MoE layer: after attention, each token has a vector representation.
Router scores the experts: the gating network produces one score per expert (512 scores in Qwen3.8-2.4T-A95B, 896 in Kimi K3, 192 in Tencent Hy3).
Top-k selection: only the k best-scoring experts are kept: 10 routed plus 1 shared in Qwen3.8, 16 routed plus 2 shared in Kimi K3, 8 in Hy3, 2 in Mixtral 8x7B.
Experts compute in parallel: each selected expert processes the token. On multi-GPU servers experts are often spread across devices ("expert parallelism"), which is why serving guides mention options such as expert-parallel flags.
Outputs are combined: expert outputs are mixed using the router weights and the result goes to the next layer.
Training keeps the load balanced: without care, a router can send most tokens to a few experts. The Switch Transformers paper describes training instability and communication cost as historic barriers and proposes techniques to address them; individual model cards rarely state their balancing method.

Variations exist. NVIDIA describes Nemotron 3 Super as a "Latent Mixture-of-Experts" in a hybrid design with Mamba-2 and attention layers, where tokens are projected into a smaller latent dimension for routing and expert computation. Qwen3.8-2.4T-A95B interleaves linear-attention (Gated DeltaNet) and gated-attention blocks, each followed by an MoE block. Mistral AI describes Mistral Large 4 as a "granular Mixture-of-Experts architecture".

Which current models use mixture of experts?

As of 2026-11-15. Figures are as stated on each publisher's own model card or documentation page; where a page gives no value the cell says so. Providers alphabetical.

Provider	Model	Total parameters	Active per token	Experts (routed selected / total)	Context	Licence
Alibaba Qwen	Qwen3.8-2.4T-A95B	2.4T	95B	10 routed + 1 shared / 512	262,144 native, extendable to 1,010,000	qwen3.8-max licence
DeepSeek	DeepSeek-V4-Pro (0813)	1.7T	Not stated by DeepSeek	Not stated by DeepSeek	Not stated on card (384K recommended max output)	MIT
MiniMax	MiniMax-M3	About 428B	About 23B	Not stated by MiniMax	1M	minimax-community
Mistral AI	Mistral Large 4 (26.10)	1.05T (plus 1.6B vision encoder)	52B	Not stated by Mistral AI	1M	Mistral Ecosystem Open Weight License
Moonshot AI	Kimi K3	2.8T	104B	16 routed + 2 shared / 896	1,048,576	Kimi K3 License
NVIDIA	Nemotron 3 Super 120B-A12B	120B	12B	Not stated by NVIDIA (LatentMoE)	Up to 1M (256K default config)	NVIDIA Nemotron Open Model License
OpenAI	gpt-oss-120b	117B	5.1B	Not stated on the model page	131,072	Apache 2.0
Tencent	Hy3 (Hunyuan)	295B (card text)	21B	8 / 192	256K	Apache 2.0

Many other open-weight publishers state total and active figures in the same way on their own cards; always take the numbers from the publisher card rather than from a third-party listing.

What do closed-model providers say about architecture?

Provider	Model page checked	Architecture or MoE status	Parameter count
Anthropic	Claude models overview (Claude Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.2)	Not stated by Anthropic	Not stated by Anthropic
Google	Gemini API models page and Gemini 3.8 Flash page	Not stated by Google on these pages	Not stated by Google
OpenAI	GPT-6.1 Sol model page	Not stated by OpenAI	Not stated by OpenAI
xAI	Grok 4.7 model page	Not stated by xAI	Not stated by xAI

Because these providers do not state an architecture, any claim that a particular closed model "is MoE" or has a given number of experts is speculation unless the provider confirms it. The Model Cards and System Cards Explained page covers what cards do and do not disclose.

Is a mixture of experts model cheaper to run?

It depends on which cost you mean. The table separates them.

Cost driver	Set by	Effect of MoE	Example from a model card
Weight storage and GPU memory	Total parameters and numeric precision	No saving: every expert must be loaded, because any token may route to any expert.	Kimi K3 must hold 2.8T parameters even though 104B are active.
Compute per token (FLOPs)	Active parameters	Large saving compared with a dense model of the same total size.	gpt-oss-120b activates 5.1B of 117B parameters per token.
Single-device fit	Total parameters, precision, context cache	Possible only when the full weights fit.	OpenAI states gpt-oss-120b fits into a single H100 GPU.
Multi-GPU communication	How experts are spread across devices	Can add overhead; Switch Transformers names communication cost as a barrier.	DeepSeek's serving example enables expert parallelism.
Throughput by serving stack	Inference engine and kernels	Varies widely.	The Qwen3.8 card states inference efficiency and throughput vary significantly across frameworks.
Hosted API price	Provider pricing	Set per token by the provider; architecture is not a published pricing input.	See Model Pricing Explained.

Mixture of experts vs dense models: how do they compare?

This is a factual comparison of properties, not a verdict on which design is better.

Property	Dense model	MoE model
Parameters used per token	All	A subset (active parameters)
Numbers on the card	One size	Total and active (Nemotron 3 Super 120B total, 12B active)
Memory to host	Proportional to total size	Proportional to total size
Compute per token	Proportional to total size	Proportional to active size plus routing overhead
Extra components	None	Router, load balancing during training, often expert parallelism at serving time
Fine-tuning	Standard tooling	Needs tooling that supports the specific MoE layout; check the model card's supported frameworks

How to read MoE figures when choosing or hosting a model

Factual criteria only; the right answer depends on your workload and hardware.

Read total and active separately: size your GPU memory from total parameters and precision (BF16 uses 2 bytes per parameter, so 120B parameters need roughly 240 GB for weights alone, before the context cache).
Check the stated context default: NVIDIA's Hugging Face config for Nemotron 3 Super defaults to 256K even though the model supports up to 1M; longer contexts need more cache memory. See Context Windows Explained.
Confirm the licence on the card: licences range from Apache 2.0 and MIT to provider-specific licences (Kimi K3 License, Mistral Ecosystem Open Weight License, NVIDIA Nemotron Open Model License, minimax-community). See Open-Weight vs Closed Models Explained.
Check supported serving engines: cards name the engines they test (Qwen3.8 lists SGLang, vLLM and TokenSpeed). Pilot on the stated engine before committing hardware.
Do not compare quality by active size: active parameters describe cost per token, not capability. Use published benchmarks with their sources; see How to Read LLM Benchmarks.
Treat hosting as a security decision too: self-hosting large MoE weights moves patching, access control and model supply-chain checks to you. See Self-Hosted LLMs vs Hosted LLM APIs and Running LLMs Locally Explained.

Where identity and security fit

Model provenance: MoE weights are large multi-file downloads. Verify the publisher's official repository and checksums, and treat third-party repackaged builds as untrusted. See the AI Data Poisoning and Backdoors Guide.
Serving credentials: multi-GPU inference clusters use service accounts, registry tokens and API keys for the model endpoint. Scope and rotate them like any non-human identity; see the LLMjacking Guide.
Application risks are unchanged: prompt injection and data leakage apply to MoE and dense models alike. See the LLM Security Guide.

Common misunderstandings

"Only the active parameters need memory": false; every expert must be resident for routing to work.
"Each expert is a topic specialist": no card cited here claims human-readable topic specialisation; specialisation emerges from training.
"MoE means fewer parameters": MoE models usually have more total parameters than dense models with similar compute per token.
"A closed model is MoE because others are": Anthropic, Google, OpenAI (GPT-6 series) and xAI do not state architecture on the pages checked.
"Total size in the name is exact": card text and repository metadata can differ (Tencent Hy3 card text says 295B, metadata 299B; NVIDIA's page meta description says 124B for the 120B model). Quote the card text and note the source.

Practitioner checklist

Record total parameters, active parameters, experts and selected experts from the publisher's own card, with the date checked.
Size GPU memory from total parameters, precision and the context length you will actually serve.
Confirm the licence text on the card before deployment.
Pilot on a serving engine the card names and measure throughput on your own prompts.
Write "Not stated by provider" for any architecture detail a closed provider does not publish.
Download weights only from the publisher's official repository and verify integrity.
Scope and rotate the service identities that serve or call the model.

Mixture of Experts Explained: How Sparse LLMs Work and Why It Matters