The SGLang team and Ant Group's Ling Infra team published a joint tuning writeup on August 21: running Ling-3.0-flash on four Blackwell GPUs, they cut the time-per-output-token (TPOT) at batch size 1 from 3.33 milliseconds to 1.53 milliseconds — a 54% drop — pushing throughput from 288 tokens per second to 606. After swapping in DSpark, the draft model they trained for speculative decoding, average TPOT fell further to 0.78 milliseconds and throughput reached 1,120 tokens per second, with an average of 9.95 draft tokens accepted per step.
Ling-3.0-flash is a hybrid-linear-attention MoE model: of its 42 layers, 35 use KDA linear attention and 7 use MLA full attention, with 512 routed experts, a hidden width of 2,560, a vocabulary of roughly 157,000 tokens, and about 63GB of weights per card in bf16. The test setup split the model four ways with tensor parallelism, running in bf16 precision.
The GPU Is Actually the Idle One
This optimization targets the batch-size-1 case specifically. Most public inference-optimization work chases higher total throughput at high concurrency, because that's what shapes a cloud provider's bill. A different workload feels more immediate: batch size 1 — running a model locally, code completion, an agent working through one long serial chain. Only one request is in flight at any moment, and a user is watching the tokens appear on screen.
When concurrency drops, the GPU goes idle. Each decoding step does so little computation that the GPU finishes its work and sits waiting for the next batch of instructions — and those instructions come from the CPU. The moment anywhere on the host path needs to read a value back from device memory before deciding what to do next, the entire pipeline stalls to wait for that synchronization to finish.
"at batch 1, look for blocking reads of device values on the host path before anything else, because each one converts the entire host loop from hidden work into a GPU bubble."
At batch size 1, look first for blocking reads of device-memory values on the host path — each one turns host-side overhead that would otherwise stay hidden into a visible GPU bubble.
Removing the Waits One by One
The first category is host idling. The fix removes GPU synchronization points at each step: the KDA attention backend is told it never needs to read sequence length back to the CPU, so the relevant indices stay in device memory the whole time. That lets the instructions for step k+1 go out before the host even knows how many draft tokens step k accepted — the host no longer queues up waiting for a result.
The second category is shortening the work sitting on the GPU's critical path. Programmatic Dependent Launch (PDL) is used to issue weight loads early whenever they don't depend on the previous kernel's output. Routing and MoE computation, which used to take two separate kernel launches, are merged into one — with 512 experts, launch overhead alone adds up. The precision of the router gate and lm_head computations is switched from fp32 to bf16, which alone accounts for roughly a 10% gain. KDA's loop state under speculative decoding is changed to a staged update that only commits after verification passes.
All of these changes have been merged into SGLang: a fused metadata graph capture, a fused KDA verification kernel, a host-issued FlashInfer plan (which removes a blocking device-memory readback), and a scheduling order that merges the draft, verify, and extend steps into one pass.
Converting to Human-Perceptible Time
Do the rough math for a 5,000-token response. At 3.33 milliseconds per token, it takes 16.6 seconds to finish; at 1.53 milliseconds, 7.7 seconds; at 0.78 milliseconds, 3.9 seconds. Same machine, same model, same person waiting — the difference goes from "long enough to make tea" to "finished before you're done talking." That gap doesn't come from a better GPU; it comes from clearing out host-side waiting.
The key number for the DSpark tier is an average acceptance length of 9.95. Speculative decoding works by letting a small model guess a run of tokens first, which the large model then verifies in one pass — whatever it guesses right is free. An acceptance length near 10 means the large model gets through nearly ten rounds of ordinary decoding for every single verification pass. That number traces directly back to how the draft model was trained: DSpark's draft model is distilled from Ling-3.0-flash's post-training output distribution, with a loss term specifically added to optimize for acceptance length. Guessing accurately here was trained in, not luck.
An Upstream Collaboration
The writeup carries three-way credit: RadixArk on the SGLang side, Ant's Ling Infra team, and Ant's inclusionAI. Model weights and reproduction commands were released alongside it, including the DSpark variant.
A Chinese open-weight model placing well on leaderboards isn't news anymore. What's less common is having a say at the inference-stack layer — this optimization didn't wait for an upstream framework to catch up; it brought real hardware costs and real workloads to the upstream code, changed it, and merged the change back. For anyone deploying on the same framework downstream, these switches just work once flipped, regardless of whose model is running.
Sources: SGLang official technical blog, CocoLoop, Ant inclusionAI's public model card; TPOT, throughput, and acceptance-length figures were verified against the reproduction commands and benchmark results in the blog post, all measured at batch size 1, 4-card tensor parallelism, bf16.