Ant Group and vLLM open-source Humming quantization kernels

Ant Group and the vLLM community have jointly open-sourced Humming, a quantized inference kernel library. The code lives under the vLLM project organization and can be installed directly with pip install humming-kernels. Starting with vLLM 0.29.0, the CUDA build of the package already bundles this dependency, so users can switch over by adding --linear-backend humming or --moe-backend humming.

The blog post lists Jinzhen Lin, Zekai Gu and Zhilin Huang as Ant contributors, alongside community members including Michael Goin of Red Hat.

One library for a long list of formats

The long-standing headache in quantized inference is the sheer number of formats. 4-bit weights with 16-bit activations call for one set of kernels, 8-bit activations call for another, and FP4, MXFP4 and NVFP4 each have their own implementations. Humming's approach is to separate how a quantization scheme is described from how the kernel executes it, then assemble concrete kernels from shared components. In the blog's own words, it is "separating quantization descriptions from execution choices".

Supported combinations currently include WnA16, W8A8, W4A8 and W4A4, with data types ranging from INT1 to INT8 and FP4 to FP8, plus MXFP4, MXINT4 and NVFP4. On the hardware side, it covers every NVIDIA architecture from Turing onward, from Hopper parts such as the H20 and H200 to Blackwell's B300 and the consumer-grade SM120.

Several single-layer matrix-multiplication comparisons follow (all self-reported by the project team; see the charts in the original post for test conditions):

The end-to-end test used DeepSeek-V4.1-Flash on eight H20 GPUs with tensor parallelism of 8. After the MoE portion was switched from Marlin W4A16 to Humming W4A8, output throughput improved by roughly 14% to 43% and time to first token fell by roughly 25% to 32%, with larger gains at higher concurrency.

What it means for deployers in China

The blog's test platform looks almost tailor-made for the reality of inference clusters in China. The H20 is the NVIDIA card built for the Chinese market: compute is cut down while memory bandwidth is retained, which makes low-bit quantization close to mandatory when serving large models. The end-to-end test also uses DeepSeek's own MoE model, and that combination is exactly what many Chinese teams have on hand.

Another detail is support for CUDA-compatible domestic hardware. The blog lists Alibaba T-Head's Zhenwu 810E among supported devices, and the roadmap says more T-Head PPU devices, such as the Zhenwu M890, will be targeted. Ant and T-Head both sit within the Alibaba ecosystem; how far this line can go depends on the pace of upstream merges.

For teams already running vLLM, the migration cost is mostly validation. Formats that Humming can read directly do not require re-quantizing the model. Formats vLLM does not natively support, such as GPTQ INT3, can go through the compatibility path via --quantization humming.

The limitations are spelled out

The blog lays out its constraints plainly. The new instructions on the B300 no longer support INT8 compute, so INT8 matrix multiplication has to use the older instructions and performs noticeably worse. The next-generation Rubin does not support the two hidden Tensor Core formats found on Blackwell. Kernel configurations are currently chosen by heuristic rules, which the authors acknowledge cannot find the global optimum; auto-tuning is not finished.

Compilation speed is another selling point. On an H200 with caching disabled, a single W4A16 kernel compiles in just 1.88 seconds using Humming with NVRTC, and during model warm-up the kernels covering every size of a single layer take roughly 2 to 4 seconds to build. For teams that frequently swap models and formats while experimenting, that saves more time than the throughput figures do.

Sources: vLLM official blog, Humming GitHub repository, CocoLoop; throughput, latency and compilation times follow the test methodology published by the project team.