saved
Ori Eval: Find the Best Model for Your Project
gist
OpenRouter's Ori Eval is an agent that writes and runs evaluations against your codebase so you can pick a model with scored proof instead of guesswork. It finds where models already run, pins the harness and provider so later scores stay comparable, and checks the tools an agent called, the tools it avoided, and answer quality. Evals are TypeScript files executed with bun test. They can use your gold data, fail a separate CI job, and be re-run when new OpenRouter models appear.
ideas
- Pick the model for this project. There is no universal winner; Ori Eval compares models against your surfaces, budget, and gold answers.
- Pin the harness or the score moves. Ori pins model and effort during a run so a later difference comes from the agent, not a swapped provider.
- Evals are ordinary TypeScript tests. After an interview it writes .eval.ts files and runs them with bun test; CI can fail the build.
- Grade tools and answers. An eval asserts tools called, tools avoided, and answer quality; open-ended answers use an LLM judge and a minimum score.
- Use your data and a price ceiling. Compare new OpenRouter models to the incumbent on every gold row, and keep the current model in the mix if you want.
quotes
“Ori Eval helps you find that model, with quantitative proof.”
“Ori Eval's agent harness and model are pinned, so every run scores the same setup.”
“The majority of harnesses that people use are limited to the company's AI models.”
“An eval checks three things: the tools the agent called, the tools it avoided, and the quality of the answer”