On October 20, DeepSeek open-sourced DeepInfer, the large-scale inference framework it uses internally, on GitHub under the Apache 2.0 license. The repository's front page describes it as a high-performance inference framework for DeepSeek models. The open-source version provides the core features needed for offline inference and large-scale serving, and it currently supports just one model: DeepSeek-V4.1-Flash.
Public reports add that DeepSeek also open-sourced a token encoding/decoding component called deepseek-recipe, and say that on this framework a single V4.1 Flash cluster reaches a throughput of 155 trillion tokens per day. That figure comes from the company itself: the performance section of the repository README shows only throughput and latency curves with no specific values, and no third party has reproduced the number so far.
What is in the repository
The capabilities listed in the README fall roughly into four groups.
The first is kernels and runtime. It bundles FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, plus a set of custom kernels written in TileLang. DeepSeek released the first few of these during its open-source week last year and in several later releases; this time they are packaged into a shell that can start serving directly.
The second is parallelism. Attention uses data parallelism combined with balanced context parallelism, while the MoE layers use expert parallelism with load balancing (EPLB).
The third is speculative decoding, which builds on DSpark, the framework DeepSeek open-sourced with Peking University at the end of June, along with a confidence-based verification scheduler. The default draft length is 5.
The fourth covers engineering details: CPU-side batch scheduling overlapped with device execution, enabled by default; sliding-window attention that replays only a bounded suffix, saving part of the prefill computation; and structured output with regex, JSON Schema and EBNF grammar constraints.
The test environment is spelled out in detail: 4 SM100 GPUs for prefill and 64 for decode, all within the same scale-up domain. The scripts to reproduce the setup are published alongside.
Two hardware platforms, unequal capabilities
Two hardware targets are listed: Nvidia SM100-series GPUs and Huawei Ascend 950 NPUs. The requirements are CUDA 13 and CANN 9.2 respectively, plus GCC 13 or later and Python 3.12.
Two capabilities, however, are marked "NVIDIA only." One is batch-invariant inference, where a given request produces the same output no matter which other requests are batched with it. The other is deterministic sampling, where identical inputs and sampling parameters yield identical results. Both are useful for teams running evaluations and regression tests, and for now they are available only on Nvidia cards.
The README is also candid about what is missing. The KV cache system, the three-way split into vision encoding, prefill and decode, and load balancing across data-parallel replicas are not part of this release. The repository says most of the engine features needed for disaggregated deployment are already implemented, but anyone who wants to build a service like DeepSeek's production setup will have to fill in these layers themselves.
Reading the open-source releases together
DeepSeek has been releasing its infrastructure piece by piece. In April last year it published a note in the open-infra-index repository saying it would not open-source the full code of its internal inference engine, and would instead contribute design improvements and standalone components to the community, for example to existing projects such as vLLM.
The cadence since then has largely followed that approach: the kernel libraries came out on their own first, the speculative decoding framework DSpark followed at the end of June, and in September DeepSeek published DSec, the sandbox foundation for agent training, on arXiv. DeepInfer is the first time these parts have been assembled into a framework that can be deployed on its own, which opens the door wider than the "components only" position of more than a year ago.
For Chinese companies deploying V4.1 Flash privately, there is now an official reference implementation. For community frameworks such as vLLM and SGLang, DeepSeek's kernel optimizations may increasingly land in its own framework first and be ported by the community afterward. The repository passed 3,000 stars on its launch day, and the issue tracker and independent benchmarks to come will say more than the official curves about how it performs in other people's data centers.
Sources: the DeepInfer README in DeepSeek's GitHub repository (used to verify supported models, hardware, Nvidia-only features, test GPU counts and license), CocoLoop, DeepSeek's open-infra-index documentation; the single-cluster throughput figure is the company's own claim and has not been independently reproduced.