saved
So you want to use OpenRouter?
Hraness cites a source capture. The source author remains the source.
gist
Mo Moustafa runs Olly, an iMessage AI assistant, through OpenRouter at multi-million-message scale and catalogs provider-level pitfalls that model names hide. The same open weights can swing ~20 points on tool-calling benchmarks across hosts, vision endpoints can silently fail while returning 200, effort knobs and quantization filters are unreliable proxies, and agent loops must parse raw tool markup, treat empty 200s as failures, and normalize per-provider reasoning history. Pinning a short provider list still collapses under rate limits and disappearing endpoints.
ideas
- Model id is not the model you get. OpenRouter routes one slug across many hosts with different precision, parsers, and bugs; paper-identical weights behave differently in production.
- Score the board for your workload. Per-provider GPQA and TAU gaps are large; tool-calling swings of 20 points matter more for agents than headline knowledge scores, and rankings reshuffle across models.
- Declared precision is a bad quality filter. fp4 hosts sit mid-pack with fp8, worst hosts appear in every bucket, and hard quantization filters shrink fallbacks without buying quality.
- Treat hollow 200s and missed parsers as failures. Empty content/reasoning, missing usage, and tool calls left in raw XML need client-side parsing, throws, and retries.
- Do not trust a pinned shortlist forever. Even three reliable providers can all 429 or stop serving the model; benchmark from prod IPs and keep real fallbacks.
quotes
“They're the same model on paper, but very different models in real life.”
“For an agent TAU is the score that matters and a 20 point swing is not noise.”
“A 200 tells you the request was served, not that there's an answer in it.”
“So the contract isn't per model, it's per provider.”