Locals: run many AI agents at once on your Mac

gist

Locals is an MIT-licensed Python server from Generated Computers that runs MLX models on Apple silicon for many coding agents at once, up to 3x faster than mlx-lm. On an M2 Pro with 16 GB and Qwen3.5-4B, eight requests at once reach 271 tokens a second against 87 for mlx-lm, and eight agents get a first token in 1.6 s against 42 s on Ollama. It builds on Arip Asadulaev's paper Breaking the Space Barrier, which checks tool calls and JSON against a grammar on the GPU in about 3 MB.

ideas

• Batching built for a few requests at a time. Requests from every agent are computed together with GPU kernels tuned for small batches, so mlx-lm's plateau (87 tokens a second for both four and eight requests) becomes 160 and 271; on an M2 Max, Qwen3.8-27B goes from 25 to 78 tokens a second.
• Tool calls that always parse. Replies follow each model's own tool-call format under a GPU grammar, so agents call only offered tools with typed arguments; text the format already decides is written without running the model, and sixteen tool-calling agents finish in 29 s against 75 s.
• Conversations persist between turns. Locals keeps each conversation and the 3,000 to 16,000-token agent system prompts on disk across restarts, so the next turn of a 138,000-token conversation starts after 2.4 s and a first request after restart takes 1.7 s against 16 s.
• One command per agent. locals code opens OpenCode, and locals launch connects Hermes Agent, OpenClaw, Claude Code, Kilo Code, Cline, OpenHands, and Aider; it also answers OpenAI, Anthropic, and Ollama APIs on port 8321.
• The gains have stated limits. A single request with nothing to reuse runs about as fast as mlx-lm (64 against 61 tokens a second), prompt-bound work stays GPU-limited, non-standard quantization runs at mlx-lm speed, and the M5 Pro gain is about 1.5x because MLX already uses its neural accelerators.

quotes