Resin: building blocks for LLM inference engines

gist

Eric Zhang's Resin is an experimental library of parts for large language model (LLM) inference, not a finished engine. A Rust core handles batch scheduling, the key-value (KV) cache and metrics, while you write the model's forward pass as kernels tuned for your hardware. The README says this let agents build DeepSeek-V4.1-Flash and GLM-5.3-Flash engines for Blackwell GPUs in 12 hours. The Python and Rust repository, created August 8, 2026, has 21 stars, no releases and an Apache 2.0 license.

ideas

• The engine plans; the model runs. The Rust Engine turns queued requests into a frozen BatchPlan, a Python Runner returns the next tokens, and the engine commits them. A custom runner needs only a forward(plan) method.
• One cache pool serves every attention shape. A flat slab pool holds full-history KV, sliding-window and recurrent-state groups, with opt-in prefix caching that shares immutable pages and copies window pages on write.
• Agents are the intended builders. Two bundled skills cover writing an engine for a new model and an "autoresearch" loop for tuning it. Charts put DeepSeek V4.1 Flash prefill and decode within 5 to 10 percent of DeepInfer.
• Serving ships in the box. A native HTTP server speaks the OpenAI Chat Completions and Responses and Anthropic Messages APIs and exports Prometheus metrics. The Modal deployments skip proxy auth, so anyone with the URL can spend GPU time.
• The author steers production users elsewhere. The README sends anyone who needs a working inference engine to SGLang or vLLM.

quotes