The LMSYS team has published a set of serving-side optimization results for DeepSeek-V4-Pro, benchmarked on Nvidia's H20 against the flagship B300 as the baseline. The work is credited to Tianyu Zhang, Yusong Gao and Yun Zhang, and all of it runs on the open-source SGLang framework.
Two headline numbers stand out. In prefill, a PP2-CP8-TP8 configuration hits 16,900 tokens per second at 4K context; switching to PP4-CP8-TP8 for a 512K long context actually pushes throughput higher, to 25,860 tokens per second, processing 1 million tokens in 43.7 seconds. In the low-latency decode scenario, single-batch throughput on H20 reaches 271 tokens per second versus 383.7 on B300 — a ratio of 1.42x.
A separate configuration set targets high-throughput scenarios. DP32-EP32 at 4K context with 32-way concurrency delivers 703.15 tokens per second per GPU; DP16-EP16 does slightly better, at 759.73. Even at 1 million tokens of context, per-GPU throughput still holds at 66 to 67 tokens per second.
What 1.42x actually means
H20 is Nvidia's compliance-restricted variant built for the China market, and on paper its compute is well below the B300's by more than a small margin. Narrowing the single-batch decode gap to 1.42x suggests the bottleneck isn't peak compute — it's memory bandwidth, communication overhead, and scheduling idle time.
That also explains what dominates the optimization list. The biggest wins don't come from operator precision at all: fusing the quantization hot path — folding the SwiGLU activation and quantization together and dropping the intermediate buffer — lifts single-item throughput by 44.0%; replacing the vocabulary weight dot product with a transposed GEMM cuts redundant reads under high concurrency and adds another 22.8%; and rebalancing expert placement based on measured expert preference contributes a further 13.5%.
The prefill side follows the same logic. Fusing seven operators into three execution groups cuts first-token latency by 3.5%; pulling high-frequency shapes from real production routing histograms and tuning both sets of expert weights specifically for them cuts first-token latency by another 11.35%. That last move is notably pragmatic — it doesn't assume an even request distribution, it tunes for the skew that production traffic actually has.
There's also a structural swap: during prefill, tensor parallelism replaces expert parallelism for splitting the MoE layer. The reasoning is that uneven expert load drags out long tails, and it's worth paying a bit more in communication to cut that tail off.
Capacity is traded, not given
A harder constraint than speed is capacity. Two of the techniques here target memory directly: Humming MXFP4AFP8 stores expert weights in MXFP4 paired with online FP8 activations, gaining 1.71x to 4.47x capacity over baseline; online C128 KV compression maintains a compact aggregated state instead of an index-by-index one, contributing 2.268x. Stacked together, the range comes to 3.88x to 10.14x.
For deployers, this is the easiest number to translate into cost. On the same 8-GPU box, the concurrency or context length it can hold multiplies several times over, and cost per token drops by roughly the same ratio. When you can't get more chips, capacity multiplier is the one variable still left in your own hands.
DSpark's contribution is broken out separately: coordinating target execution and verification steps across pipeline stages improves per-token output time by 74.8% to 78.0%.
Mapped onto cost, the picture gets more concrete. Assume a fixed monthly cost of ownership for one 8-GPU H20 box: a 4x capacity gain means the same bill now serves 4x the concurrency, pushing the machine cost per token down to roughly a quarter (a rough estimate that ignores the extra compute long context adds and routing jitter). For services priced per token with margins already squeezed by price wars, this kind of headroom is often more real than waiting for a new generation of chips — especially when a new generation of chips isn't guaranteed to be available at all.
What it hands over is a configuration table
What this work ultimately delivers isn't a single best setting, but a configuration table sorted by scenario: for prefill, use PP2 under 32K and switch to PP4 above 128K; for decode, go PP2-TP8 for low latency, DP32-EP32 for throughput, with DP16-EP16 kept as an efficiency reference.
The format itself is a signal. A leaderboard wants one number; a production environment wants to know what its own traffic looks like and which row to land on. For teams stuck with only H20 and still on the hook to serve a 1.6-trillion-parameter model, this table is more directly usable than any single peak-performance record.
Sources: LMSYS technical blog, CocoLoop, SGLang project documentation. Throughput, latency and capacity multiplier figures are drawn from the configuration tables in that blog post; B300 comparison data comes from the same source.