On May 20, NVIDIA's Nemotron Labs released a paper and model weights for a new decoding method called Nemotron-Labs-Diffusion, available in 3B, 8B, and 14B sizes with base, instruct, and vision-language variants.
The key innovation is not parameter count but support for three decoding modes within a single model.
Three modes in one model
Traditional large language models generate text autoregressively (AR), producing one token at a time from left to right. Diffusion models predict a batch of tokens in parallel, boosting throughput but often lagging in accuracy. Nemotron-Labs-Diffusion combines both approaches and adds a third mode:
- Autoregressive (AR): standard left-to-right generation, 1× baseline speed
- Diffusion: denoises multiple tokens at once, 2.57× tokens/forward
- Self-Speculation: diffusion drafts candidate tokens, AR verifies them, 5.99× tokens/forward
Self-Speculation is the core innovation. The diffusion part drafts a batch of candidate tokens in parallel, then the AR part quickly validates which to accept. No extra small draft model or prediction head is needed—the same model and weights serve both modes. On the 8B model, Self-Speculation achieves 5.99× tokens per forward compared to Qwen3-8B, with accuracy slightly higher (63.61% vs. 62.75%).
Measured performance on GB200
Throughput numbers on a single GB200 GPU with concurrency=1: 850 tokens/s vs. 253 tokens/s in AR mode. With NVIDIA's custom CUDA kernels, it reaches 1015 tokens/s. The 8B vision-language variant achieves 3.63× to 7.45× tokens/forward depending on response length. For context, mainstream LLM inference on H100 typically runs at 100–300 tokens/s. Nemotron-Labs-Diffusion 8B on GB200 runs at 1015 tokens/s—3–4× faster than leading models. This matters for real-time agents that need to reason, call tools, and continue reasoning. Tripling throughput can cut agent latency from 15 seconds to 5 seconds, a key threshold for moving from demo to daily use.
Training approach
Nemotron-Labs-Diffusion is not trained from scratch. It adds a second phase after AR pretraining:
- Phase 1: 1 trillion tokens, pure AR objective
- Phase 2: 300 billion tokens, mixed AR and diffusion objectives
The combined loss function is ℒ(θ) = ℒ_AR(θ) + α · ℒ_diff(θ), with α = 0.3. This means AR is the primary objective and diffusion is auxiliary. This weight choice is a fundamental difference from other diffusion LMs, which treat all token orderings equally. Nemotron-Labs-Diffusion leverages the natural left-to-right prior of language, avoiding the accuracy degradation often seen in diffusion LMs. Training used 256 H100 GPUs for all three model sizes.
Competitive positioning
Nemotron-Labs-Diffusion directly competes with Qwen3-8B and Llama 4-8B, but its pitch is not smarter models—it's the same intelligence at 3–6× throughput. NVIDIA's strategy is clear: sell more GB200s and this model to extract more token economics than competitors. This aligns with NVIDIA's year-long shift from selling chips to selling a full stack (model + hardware + CUDA kernels) that makes enterprise compute bills more attractive. The weights are available on Hugging Face under the NVIDIA Nemotron Open Model License, which permits commercial use, fine-tuning, and deployment. NVIDIA has not disclosed whether the method scales to 32B/70B models, but the May 20 release is enough to keep inference optimization teams busy discussing it tomorrow.
Sources: NVIDIA AI Releases Nemotron-Labs-Diffusion: A Tri-Mode Language Model with 6× Tokens Per Forward Over Qwen3-8B (MarkTechPost); Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding (NVIDIA Research); NVIDIA pushes past autoregressive text generation with Nemotron-Labs-Diffusion (Startup Fortune); CocoLoop