Overview: Mixture of Experts

Justy Okay, Cody, we have dodged this one long enough. We keep saying MoE like everybody in the room already got the memo, and I think we owe the concept an actual explanation.

Cody Yeah. Every time we name-drop it, we do that thing where we act like it’s obvious and then immediately move on. Which is very on-brand for us and also kind of rude to the idea.

Justy It’s such an Exploring Next move. We mention it, nod like we’ve done the work, and then the poor listener is left holding the bag.

Cody And honestly, the idea itself is pretty elegant once you stop pretending the whole model has to think about everything. The clean mental model is this: don’t wake up the whole brain for every word. Route the word to a couple of specialists.

Justy Right, that already makes sense to me. So instead of one giant pile of weights doing everything, you’ve got a bunch of smaller experts, and some little chooser says which ones matter for this token?

Cody Exactly. Think of a neural network as a machine with lots of learned knobs and connections. A neural network is just a system that learns patterns by adjusting those connections from data, instead of us hard-coding the rules.

Justy Mm-hm.

Cody Now a transformer is the kind of neural network behind a lot of modern language models. It processes text in layers, and each layer helps the model refine what the next token should be. We don’t need the full transformer zoo here, just that basic stack of layers.

Justy And the famous attention part?

Cody Attention is the mechanism that lets the model look back at the most relevant pieces of context while it’s working on a token. It’s like the model keeps a little spotlight on what matters right now. MoE usually sits alongside that inside the model, not instead of it.

Justy Okay, so the brain is already layered and already looking around. Where does Mixture of Experts actually enter the picture?

Cody In the feed-forward parts of some transformer layers. Feed-forward just means the chunk that takes the current representation and transforms it without the back-and-forth lookups attention does. MoE swaps a single big feed-forward block for many smaller ones, which we call experts.

Justy So one giant room becomes a hallway of smaller rooms?

Cody Yeah, that’s a good picture. And the model doesn’t walk into every room every time. A router, which is a tiny learned model inside the bigger model, looks at the token and picks a small set of experts to activate.

Justy Wait, when you say activate, you mean actually run those weights for that token.

Cody Right. The router is making a compute decision. Conditional computation means the model only spends work on the parts that are relevant for this input, instead of running all parts every time. That’s the whole trick.

Justy That’s the part that feels almost too good. You get a giant model, but you don’t pay giant-model cost on every token.

Cody That’s the promise, yes. The important distinction is active parameters versus total parameters. Total parameters is the full size of everything stored in the model. Active parameters is the slice actually used for one token.

Justy Right, right.

Cody So with an MoE model, the headline number can be huge, but the per-token compute can look much smaller because only a subset of the network wakes up.

Justy Can you make that concrete, because otherwise it sounds like marketing math and I get nervous.

Cody Totally fair. Qwen3-235B-A22B is the clean example here. It has two hundred thirty-five billion total parameters, but only about twenty-two billion active parameters per request. And at each of its ninety-four layers, the router picks eight experts out of one hundred twenty-eight for each token.

Justy Mm-hm.

Cody So the model has the capacity of something enormous, but the per-token work is closer to a twenty-two-billion-parameter model. That’s why people get excited. It’s not that the model is magically smaller. It’s that it’s selectively smaller at runtime.

Justy Okay, that part actually clicks. It’s like having a giant specialty shop with one cashier directing you to the right aisle, instead of stocking every aisle for every customer who walks in.

Cody Yeah, and the cashier analogy is better than most because the cashier is doing a real job, not just waving at the shelves. The router has to make a learned choice about which experts are likely useful.

Justy So the experts are the shelves, and the router is the cashier. I’m going to overuse that now, by the way.

Cody Please do. It’s the clearest version we’ve got. And just to keep the jargon honest, the reason this exists at all is sparsity. Dense networks keep lots of connections active at once. Sparse networks keep fewer active, which cuts compute and memory.

Justy So MoE is basically sparsity applied inside the model.

Cody Exactly. More specifically, it adds sparsity to those linear or feed-forward layers. That’s where the model can afford to route instead of brute-force everything. The rest of the transformer still does transformer things.

Justy ‘Transformer things’ is very us, unfortunately.

Cody I know. But it’s not wrong. And it matters because the MoE layer is where the system gets its weird economics. You’re storing a lot of expert capacity, but for one token you only pay for a few of them.

Justy Which is why people keep saying these huge frontier models are MoE models now. They’re not all doing the same amount of work per token even if the parameter count is massive.

Cody Yep. And the size threshold matters. MoE tends to make the most sense at large scale, often around one hundred billion parameters and up. Below roughly thirty-two billion, and especially below eight billion, dense models are usually simpler and often just better value.

Justy That’s useful. So this isn’t the default answer for every model, it’s the answer when scale gets awkward enough that sparsity starts paying rent.

Cody That’s a good way to put it. For a small model, the overhead of routing and expert management can be more trouble than it’s worth. If you only need one domain, like a narrow completion model, the whole thing starts to look like one expert anyway.

Justy Right, because the specialization isn’t buying you much if the problem space itself is narrow.

Cody Exactly. The win shows up when the model needs lots of capacity but you still want per-token efficiency. That’s why these models are attractive for local inference too. If you’re running a very large model on a high-end machine, the low active-parameter count can make it feel much more feasible than the total size suggests.

Justy Okay, now I want to hear the catch, because there’s always a catch and I know you’ve been waiting to be annoying about it.

Cody I have been waiting, yes. The catch is that the savings are cleanest for one request at a time. In production, if you batch a bunch of requests together, different tokens can route to different experts. Across a big enough batch, you can end up lighting up almost the whole set of experts anyway.

Justy So the sparsity kind of melts when the server gets busy.

Cody Partly, yes. You still benefit from the architecture, but the exact savings people imagine from the single-request story don’t automatically survive batching. The system has to be designed around that reality.

Justy Okay, that’s the production reality part I wanted. Because if the whole pitch is ‘we only use a little bit of the model,’ I need to know whether that still holds when ten people hit it at once.

Cody And the answer is: not by default. Which is why MoE deployment adds another layer of parallelism called expert parallelism, or EP. That means you shard whole experts across GPUs. For example, if you have one hundred twenty-eight experts across eight GPUs, you can park sixteen experts on each GPU.

Justy Mm-hm.

Cody The router is small, so you replicate it on every GPU. Then each GPU figures out which tokens need which experts, and sends tokens to the GPU that owns that expert. That tends to need less communication than splitting every matrix across every GPU the way tensor parallelism does.

Justy So EP is more about throughput than making one token faster.

Cody Exactly. It helps the server handle more simultaneous tokens. A single token doesn’t necessarily get faster, but the system can do more work overall. In practice, real deployments often mix tensor parallelism for the attention parts and expert parallelism for the sparse MoE layers.

Cody You do now. That’s the unfair part.

Justy Okay, Cody, I cannot believe this is how we spent a Wednesday, but I’m weirdly happy we finally did it.