DeepSeek V4.1 Flash Large-Scale Inference Deployment Practice

gist

DeepSeek describes the production inference system behind V4.1 Flash and open-sources it as DeepInfer, in an X Article that Latent.Space translated from Zhihu. One cluster serves 155 trillion tokens a day at about $38 per card-hour off-peak, using separate vision, prefill and decode tiers, wide expert parallelism on 64 to 128-card decode supernodes, and KVCache kept for 72 hours or more. Its production table reports a 96% input cache hit rate, 400 ms average time to first token and 230 tokens per second output.

ideas

• Prefill and decode run on different hardware. Prefill uses ordinary 8-card machines at EP2 to 8, while decode uses 64 to 128-card supernodes at EP64 to 128, and a cheap single card is enough for the vision encoder.
• Cache storage is a dedicated cluster. About 100 PiB of 3FS SSD with 10+2 erasure coding holds Block KVCache, with TTLs of 3 days for agent traffic and 7 for chatbots, and peak reads of 10 TiB/s.
• Bounded replay replaces saved window state. Instead of persisting sliding-window cache, V4.1 recomputes only the last 128 tokens on a miss, and doing the same at decoder layers nearly halves total prefill compute.
• Every tier has its own load balancer. Prefill routes by a fitted quadratic attention cost in four length buckets, decode balances request count against KVCache length, and redundant experts held load imbalance near 1.03 to 1.09.
• The tokenizer layer guards special tokens. DeepSeek Recipe, written in Rust, encodes special-token text in user input as ordinary tokens and parses streamed output across chunks, while DSpark variable-length verification adds 6% throughput.

quotes