4 verified stories covering MoE, product updates and industry developments.
Meta released Llama 4 on April 5, 2025, introducing the first MoE architecture in the Llama series with two models: Scout (17B active parameters, 10M token context) and Maverick (17B active, 128 experts, 1M token context), both natively multimodal via early fusion.
Moonshot AI open-sourced the trillion-parameter Kimi K2 MoE model, activating only 32B parameters per token, with strong coding and agent performance.
DeepSeek V3.1 is the company's first hybrid model, combining the capabilities of the general-purpose V3 and the reasoning-focused R1 into a single architecture.
DeepSeek V3 uses a Mixture-of-Experts architecture with 671B total parameters but only 37B activated per token, achieving top-tier open-source benchmark scores while costing just $5.5 million to train.