On April 5, 2025, Meta released Llama 4, the first Llama series model to adopt a Mixture-of-Experts (MoE) architecture. Two variants were launched simultaneously.
Llama 4 Scout
- Active parameters: 17B / Total parameters: ~109B
- Number of experts: 16
- Context window: 10 million tokens
Yes, 10 million tokens of context. That number set a new record for open-source models at the time.
Llama 4 Maverick
- Active parameters: 17B / Total parameters: ~400B
- Number of experts: 128
- Context window: 1 million tokens
128 experts, with MoE layers and dense layers arranged alternately — experts only participate in computation in half of the layers.
Native Multimodality
Both models achieve native multimodality through early fusion — text and images are not stitched together via an external adapter but are fused and processed natively within the model. Training data exceeds 30 trillion tokens, covering more than 200 languages.
Performance
Maverick outperforms GPT-4o and Gemini 2.0 in coding, reasoning, multilingual tasks, long-context understanding, and image comprehension, and can hold its own against the much larger DeepSeek V3.1.
Scout is positioned as a more accessible option — it supports 4-bit and 8-bit quantization and can run on a single GPU.
There is also Llama 4 Behemoth (288B active parameters), which was announced but not open-sourced, as it was still in training at the time.
One detail: Meta itself calls it "open-weight" rather than "open-source." The model weights are publicly available for fine-tuning, but the complete training code and datasets have not been released.
Sources: CocoLoop, Meta AI official blog