saved
Billion-Token Scale Trace Analysis: Jev vs LLMs
Hraness wrote this summary from a saved copy of the source. Quotations are taken word for word from the source.
gist
Applied Compute describes gap analysis over billion-token RL traces: sample with an LLM to build a failure taxonomy, then classify the full corpus with cheap calibrated models. On τ³-bench banking traces, Jev is Pareto-optimal on micro-F1 versus cost at a high-recall threshold and is better calibrated than Luna, making it a practical first-pass filter despite a 32k context limit.
ideas
- Gap analyses steer training. Failure clusters feed post-training data, monitor reward hacking, and start online RL from production traces.
- Map with LLMs, reduce with Jev. Frontier models synthesize taxonomies; system-one classifiers annotate the full run in parallel.
- Jev wins on cost-recall tradeoffs. At threshold 0.20 it reaches about 85% recall and sits on the micro-F1/cost Pareto frontier; labeling 10k traces costs about $11 versus $54 for Luna and $479 for Haiku 4.5.
- Calibration matters for triage. Jev shows the lowest ECE and Brier scores among compared classifiers, so confidence thresholds are usable.
- Long traces still need compression. Jev’s 32k context forces summarization or splitting for hundred-thousand-token agent trajectories.
quotes
“We find that Jev is particularly well suited to gap analysis workloads and is pareto optimal across cost and recall at certain decision thresholds.”
“at threshold 0.20, Jev's recall rises to 85%, and it is Pareto optimal for micro F1 versus cost at this threshold (Figure 4).”
“Jev is the most well calibrated, with the lowest ECE and Brier scores (Figure 5).”