Open-source inference framework vLLM announced on its official blog on September 24 that it has added text watermarking based on the Gumbel-max method, which can be turned on with a single server startup flag. The post is credited to three authors — one from Mistral and two from Red Hat.
Turning it on is simple: add --watermark-config after the vllm serve command, specify the algorithm as gumbel, and supply a key. From then on, every piece of text the server generates carries an invisible statistical signal.
How the Watermark Gets Hidden
Every time a large model generates a word, it first assigns a probability to every candidate word, then samples one. The Gumbel-max watermark works on that "sampling" step.
Specifically: using the key, the context of the most recent few words, and each candidate word's index, the method computes a set of numbers that look completely random but can be reproduced by anyone who knows the key. These are transformed into Gumbel noise and added to the model's scores, and the highest-scoring candidate becomes the output. Mathematically, the resulting distribution is identical to that of ordinary random sampling — which is why the authors call it "distortion-free": the model isn't nudged toward particular words, a particular writing style, or a particular class of solutions.
Detection doesn't require the model weights — only the key and the tokenizer. The text is converted back into tokens, the same key is used to recompute that set of pseudo-random numbers, each token is scored and the scores summed, and the result is compared against the distribution expected for unwatermarked text to produce a p-value. Under the authors' calibration, unwatermarked text is falsely flagged about 1% of the time.
Two Sets of Numbers: Cost and Effect
On performance, the authors tested Qwen3.5-27B generating 512-token outputs on a single H100: throughput changed by -0.23% at batch size 1 and +2.03% at batch size 32, with averages across groups ranging from -1.1% to +2.0%; the authors' conclusion is that there's no consistent throughput change. To save memory, vLLM fused the pseudo-random number generation, the Gumbel transform, and the argmax step into a single GPU kernel.
On model quality, again with Qwen3.5-27B, the comparison between watermarked and unwatermarked outputs was: GSM8K 93.0% vs. 94.2%, MBPP 79.2% vs. 77.2%, IFEval 90.7% vs. 91.9% — differences the authors consider within margin of error.
The gap in detection rates is much larger. At a 1% false-positive rate, creative writing can be fully detected at around 100 tokens; for MBPP coding problems, the detection rate is only 43% even at 400 tokens. The reason follows from the mechanism: for output like code, the model is often highly confident about the next token, leaving little randomness for the watermark to work with. Short text carries a weak signal for the same reason. When text is edited by a person, the scores near the edited spots get scrambled, though the signal recovers once you move past that stretch of context.
There are two implementation pitfalls. One is speculative decoding: the authors use two separate keys — one for draft tokens and one for the target model's residual sampling — to preserve the acceptance rate, at the cost of splitting the detection signal across two keys. The other is that repeated context produces the same set of random numbers, which can trap the model in infinite repetition; the fix is to skip watermarking whenever repeated context is detected, which costs at most 0.19% throughput.
What This Means for Deployers in China
China's Measures for Labeling AI-Generated and Synthetic Content, in effect since September of last year, require generated content to carry an explicit or implicit label. For text, implicit labeling today mostly relies on metadata; statistical watermarking can be embedded directly in the text itself, and vLLM has effectively turned that into a switch — teams in China that run large numbers of self-hosted inference services on vLLM can try it directly.
But the boundaries of this approach are also clear: detection depends on a key that only whoever deployed it holds, so third parties can't verify it independently; detection rates are low for code and short replies; and the signal weakens when text is heavily rewritten by hand. Whether this can satisfy regulators' specific requirements for "implicit labeling" still depends on whether matching detection standards emerge — there's no public guidance on that yet.
Sources: vLLM official blog, CocoLoop, Measures for Labeling AI-Generated and Synthetic Content; throughput, benchmark scores, and detection rates are all as tested by the authors on Qwen3.5-27B and a single H100.