saved
Blazing fast on-device GenAI with LiteRT-LM
Hraness wrote this summary from a saved copy of the source. Quotations are taken word for word from the source.
gist
Google’s LiteRT-LM runtime runs Gemma 4 on phones, laptops, and browsers, and Google says multi-token prediction speeds decoding up to 2.2x. Without MTP, Gemma 4 E2B decodes at 52 tokens a second on an Android GPU, 56 on iOS with Metal, and up to 76 through WebGPU on a MacBook Pro.
ideas
- Drafter and model share one chip. LiteRT-LM runs the MTP drafter and the main model on the same hardware block, such as the GPU, keeping the shared KV cache and activations local; Google measured the up-to-2.2x gain on a Samsung S26 Ultra GPU.
- Memory is the real constraint on phones. Keeping per-layer embeddings out of memory and loading image and audio encoders only when needed lets the 2.58 GB E2B model run in a 607 MB physical footprint on Apple mobile CPUs using XNNPACK weight caching.
- Long context survives app restarts. Session save and restore serializes large KV cache states, so long conversations can resume without reprocessing the prompt.
- One runtime, many platforms. LiteRT-LM adds Swift and JavaScript/WebGPU APIs beside Android, plus Thinking Mode and tool use for agent tasks.
quotes
“Standard LLM inference is fundamentally memory-bandwidth bound”
“Enabling MTP in the LiteRT-LM pipeline requires only two lines of configuration”