MoE (mixture of experts)
Japanese MoE（専門家混合モデル）
MoE is a way of splitting an AI model into many small "experts" and letting only a few of them work on each input. It lets a model be large overall while using less computation at a time.
Inside an MoE model there are several small blocks of computation called experts, and a routing part chooses which experts should handle each piece of text that comes in. It is a bit like a large hospital's front desk sending each patient to only the one or two departments that fit their symptoms. Mistral AI says its Mixtral model picks 2 of 8 experts and, out of 46.7 billion parameters in total, uses only 12.9 billion at a time.
Watch out You cannot judge an MoE model's quality or cost from its total parameter count alone, because the amount it actually uses at a time is different. Also, every expert has to be loaded into memory, including the ones not chosen, so when running one on your own computer, plan memory around the full size.
Source: Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer · Mixtral of experts · Two in-house models in support of our mission (checked 2026-10-26)