vLLM Boosts DeepSeek V4.1 Flash Throughput 5.3x
vLLM's new optimizations push DeepSeek-V4.1-Flash throughput up to 5.3x under rate limits, mainly by shrinking the KV cache and skipping memory offload for long contexts.
inference optimization News and Analysis
7 verified stories covering inference optimization, product updates and industry developments.
Baseten's AI Agent Writes an Inference Engine, Beats vLLM by 90%
Baseten had a Claude Code agent generate a custom inference engine for Qwen-3.6-35B-A3B, hitting 1792 TPS on B200 GPUs — about 90% faster than vLLM 0.25.1.
SGLang speeds up MiniMax-H3 1.95x with zero quality loss
LMSYS benchmarked MiniMax-H3 on 8x H200 GPUs: SGLang's lossless path beats Diffusers by up to 1.95x, while steeper tiers trade SSIM fidelity for up to 6x speed.
Ant's Ling-3.0-flash: 2.1x Faster Single-Request Decoding on 4 GPUs
SGLang and Ant Ling Infra cut single-request TPOT for Ling-3.0-flash from 3.33ms to 0.78ms on 4 Blackwell GPUs by removing host-side stalls.
Trillion-Parameter Model Restart Cut From 8.8 Minutes to 32 Seconds
Ant Group and SGLang keep quantized weights resident in GPU memory, cutting Ling-2.6-1T restart time from 8.8 minutes to about half a minute.
H20 Runs DeepSeek V4-Pro at 70% of B300 Speed
LMSYS's DeepSeek-V4-Pro benchmarks show Nvidia's China-market H20 chip closing the decode-speed gap with the flagship B300 to just 1.42x.
NVIDIA fuses diffusion and autoregression in new Nemotron model
NVIDIA's Nemotron-Labs-Diffusion introduces a tri-mode decoding approach that boosts throughput up to 6× over Qwen3-8B while maintaining accuracy.