On August 21, Ant Group's Ling infrastructure team, together with Alibaba and the SGLang team, published a component called Weight Cache Daemon on the LMSYS blog: a process that stays resident in GPU memory, keeping quantized and sharded weights in place so a freshly started inference engine can map them in via CUDA IPC with zero copying. Their numbers: weight loading for Ling-2.6-1T FP8 dropped from about 495 seconds to 0.63 seconds, and total startup time fell from 8.8 minutes to 0.528 minutes.
Eight of Eight and a Half Minutes Was Spent Reading Disk
The team broke down a full startup run. Ling-2.6-1T FP8 running on 8 H20-3e GPUs, with weights stored on a 3.5T NVMe SSD, took about 527 seconds from launch to being ready to serve requests. Weight loading accounted for 495 seconds of that, or 93.9%; tokenizer initialization took 13 seconds, torch distributed initialization 5 seconds, CUDA graph capture 7.7 seconds, and everything else combined under 6 seconds.
Each GPU has to read roughly 120 GB of safetensors from disk, deserialize it, shard it by tensor parallelism, then run FP8 quantization and weight reordering — 161 shards in total. This pipeline has to run from scratch on every restart, yet its output is deterministic: the same model with the same configuration produces identical tensors in GPU memory every time, and often those tensors are still sitting there right after the previous process exits.
In production, those few minutes mean P99 tail latency spikes during restarts, in-flight requests either fail outright or queue indefinitely, and rolling upgrades and failure recovery are both bottlenecked by this cycle.
Keeping the Weights in GPU Memory
Each GPU runs one daemon process, one per TP rank. It loads once through the full pipeline from disk, then exports every parameter and buffer in model.state_dict() as CUDA IPC handles and hands them over a Unix socket to any engine process that connects. On the engine side, the model structure is first built on a meta device with no memory allocated, then each parameter's data pointer is pointed at the mapped-in tensor. The two processes share the same physical GPU memory with no copying at any point. Post-processing parameters generated during FP8 quantization, such as weight_scale, are cached too, so they don't need to be requantized.
Two safety gates are in place. One is a configuration fingerprint: model path, TP/PP/DP sharding, quantization method and config hash, and dtype all have to match, plus GPU compute capability and torch version are recorded as environment markers. Different architectures or torch versions take different post-processing branches — the weights could map in cleanly but produce garbage — so folding the environment into the fingerprint turns that silent failure mode into an explicit mismatch that falls back to loading from disk.
The other gate is a quantization method whitelist. Currently only no quantization and block-wise FP8 are validated; per-tensor FP8, Marlin, and AWQ/GPTQ all raise an error outright. The reason is that IPC only exports raw tensor data, while these methods keep part of their effect in Python-side metadata or reorder and transpose the weights — mapping them in directly would produce wrong values. The team chose to fail loudly rather than silently return incorrect results.
If the daemon crashes, running engines are unaffected — CUDA's reference count keeps the memory alive until both sides exit. Once the daemon restarts, it reloads from disk and re-exports handles before new engines can connect again.
A Side Effect: Solving the Hot-Standby Cost Problem
The component has three modes: daemon (the engine itself launches the daemon; the first startup is still slow), client (connects to an already-running daemon for sub-second restarts), and off (the default, reading from disk — 405 to 411 seconds for Ling-2.6-1T).
More practical than the startup speedup are a few deployment patterns it enables. Multiple engine instances on the same GPU can map the same weights, so the disk is read once and quantization runs once. A high-priority online service and a low-priority offline batch job can share a GPU, with the latter restarting in sub-second time after being preempted. In active-standby failover, the standby engine maps the same weights with zero copy and stays warm, taking over within a second of the primary instance going down.
The last point is a straightforward cost item. Rough math: a single Ling-2.6-1T instance occupies 8 H20-3e GPUs; a traditional hot standby means another 8 GPUs sitting idle alongside it. Switching to a shared-weight standby eliminates that entire idle GPU group. The same applies to rolling upgrades — at roughly 8.8 minutes per restart, each instance burns about 1.2 GPU-hours of idle time per rolling-restart cycle, and for a cluster with hundreds of instances, one upgrade round adds up to three-digit GPU-hours.
Not Done Yet
Weight Cache Daemon is phase one of the Fast Engine Recovery Framework. The roadmap targets cold starts under 10 seconds and hot-standby failover under 1 second. Next up are CUDA graph serialization, kernel caching, and distributed initialization optimization. Based on the breakdown above, those add up to roughly 26 seconds — the bulk of what's left after weight loading.
Numbers for the Qwen3-235B FP8 tier were also released: about 235 GB of weights, 306 to 327 seconds to load from disk, under 1 second via IPC mapping — roughly a 500x speedup. Ling-2.6-1T comes in at roughly 780x.
The blog also gave a nod to the newly released 2.8T-parameter Kimi K3. As parameter counts scale up, disk-loading time scales up roughly linearly, while IPC mapping stays roughly constant — a gap that only widens further. For teams already running trillion-parameter models, the cost of a restart is shifting from "wait a few minutes" to "barely wait at all," and that shift affects more than just an availability number — it also affects how willing teams are to change configs frequently, or to pack two services onto the same GPU.
Sources: LMSYS Org technical blog, CocoLoop, SGLang project documentation; the startup-time breakdown and the weight-loading vs. IPC-mapping comparisons are drawn from the team's published single-node benchmark table.