saved
llama.cpp v0.7.0
Hraness wrote this summary from a saved copy of the source. Quotations are taken word for word from the source.
gist
llama.cpp v0.7.0 adds a GPU cache for mixture-of-experts weights held in host memory, which its notes say speeds decode up to about 2.2x. It also adds LiliCorr speculative decoding on top of DFlash, multi-token prediction for GLM5-Next, lower embedding memory, and a long list of new models.
ideas
- Offloaded experts no longer pin decode to the CPU. With the --moe-cache-mib option, experts stored in host memory run on the GPU from an LRU cache, and the cache can be shared across several GPUs.
- Speculative decoding keeps getting new drafters. The release adds the LiliCorr technique on top of DFlash and MTP for GLM5-Next, so more models can draft and verify several tokens per step.
- Memory use drops for embeddings. Embedding models use up to 10x less VRAM, which matters on the same machine that runs the chat model.
- Model coverage is still the main product. New architectures include K2 Horizon, Prism Bonsai 2 27B, EmbeddingGemma-2, MiniCPM-V 4.7, and Nemotron Super 3.5 VL, and the server’s default port changes to 9931.
quotes
“up to ~2.2x decode speedup when experts are offloaded”