saved
Fast, On Device Agentic AI with Muse Glimmer on ExecuTorch
Hraness wrote this summary from a saved copy of the source. Quotations are taken word for word from the source.
gist
Meta’s ExecuTorch team runs Muse Glimmer 30B as a local coding agent on NVIDIA GPUs and Apple silicon by exporting it straight from PyTorch. With DFlash speculative decoding, the team measures text-and-image generation on an M5 Pro rising from 21.6 to 33.0 tokens a second, a 52.8% gain without quality regression.
ideas
- Export once, lower per backend. The model and its decoding strategy are written in PyTorch, then compiled ahead of time to Triton on CUDA or to MLX and custom Metal kernels on Apple silicon, optimizing the whole execution path rather than single ops.
- Draft and target ship as one file. DFlash target and draft share weights inside a single PTE artifact, with the block length chosen at runtime and both greedy decoding and rejection sampling supported.
- GGUF goes straight in. ExecuTorch exports from the released GGUF and maps Q4_K, Q5_K, and Q6_K to packed integer kernels on CUDA and repacked or fused Metal kernels on MLX.
- It is built for agents, not just chat. One model load serves several isolated conversations, with Harmony chat templates, reasoning routing, and a parser for the model’s XML tool calls; decode on CUDA is captured into a CUDA graph.
quotes
“Reimplementing these across different backends doesn’t scale.”
“Ahead-of-time compilation optimizes the full execution path end-to-end, not just individual ops.”