saved
WebLLM: high-performance in-browser LLM inference
Hraness cites a source capture. The source author remains the source.
gist
WebLLM is the MLC TypeScript engine for running open-source LLMs entirely in the browser on WebGPU, with no server. You create an MLCEngine, load a model, and call chat completions through an OpenAI-compatible API that includes streaming, JSON mode, and seeding. Workers keep generation off the UI thread, and a service worker can keep a model loaded across visits. Weights and WASM come from MLC LLM; cache backends and optional SRI hashes are configurable.
ideas
- Inference stays on the client. WebGPU runs the model in the page; nothing is sent to a server, so privacy and offline use are the default.
- The OpenAI chat surface is the integration. CreateMLCEngine loads a model, then engine.chat.completions.create accepts the same messages, streaming, JSON-mode, and seed fields.
- Workers own the model lifecycle. A dedicated web worker keeps generation off the UI thread; a service worker plus heartbeat tries to keep weights loaded across page visits.
- MLC artifacts are the extension point. A model is a weight URL plus a WASM library; variants can share a library, and custom compiles come from MLC LLM.
- Caches and hashes are explicit policy. Four cache backends and optional SRI checks on config, WASM, and tokenizer files make storage and integrity a configuration choice.
quotes
“Everything runs inside the browser with no server support and is accelerated with WebGPU.”
“You can put the heavy computation in a worker script to optimize your application performance.”
“WebLLM supports optional integrity verification for model artifacts using SRI (Subresource Integrity) hashes.”
“WebLLM works as a companion project of MLC LLM and it supports custom models in MLC format.”