Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio

gist

Apple shows MLX running Muse Spark Open, an 816B mixture-of-experts model, across four M5 Ultra Mac Studios at over 36 tokens a second. Apple credits three pieces working together: MLX support for the GPU Neural Accelerators in M5 chips, RDMA over Thunderbolt 5 between Macs, and efficient mixture-of-experts kernels.

ideas

• Neural Accelerators speed up the prompt. Time to first token is compute bound, and Apple measures the M5 Ultra up to 4.6× faster than the M3 Ultra there; decode is bandwidth bound, and 1.2 TB/s against 819 GB/s gives up to 1.6× faster generation.
• A trillion-class model fits on four desks. Muse Spark Open needs more than 1.6 TB in BF16 but about 493 GB in NVFP4, so it runs split across four 512 GB Mac Studios with tensor parallelism, reaching 36.73 tokens a second in Apple’s table.
• RDMA over Thunderbolt makes clustering practical. macOS 26.2 added RDMA over Thunderbolt 5, and MLX’s jaccl backend, launched with mlx.launch, uses it for low-latency collectives between machines.
• Pipeline and tensor parallelism trade off. Pipeline parallelism splits layers across Macs and, for 65,536-token prompts, prefills up to about 1.8× faster than one node; tensor parallelism splits each layer and speeds up decode.
• Power stays modest. Apple puts peak draw at about 200 W for an M3 Ultra Mac Studio and 300 W for an M5 Ultra.

quotes