This Microsoft research paper is f*cking mind-blowing

This Microsoft research paper is f*cking mind-blowing

Microsoft researchers just proved that a tiny 7B open-source model can beat OpenAI o1 on Olympiad mathematics without copying a single token from larger teacher models

Their breakthrough paper, rStar-Math, solves the holy grail of test-time compute: combining Monte Carlo Tree Search, step-level code verification, and a 4-round self-evolution loop

How Microsoft turned a 7B model into an Olympiad-level reasoning engine:

→ The Harness Layer (Code-Augmented MCTS): executes Python verifiers at every reasoning step to filter out invalid intermediate calculations before expanding the tree

→ The Graph Layer (Process Preference Model): uses Q-value branch comparisons to score individual reasoning steps rather than relying on noisy final-answer rewards

→ The Loop Layer (4-Round Self-Evolution): bootstraps the policy model and verifier in iterative cycles over 747k problems, unlocking autonomous self-improvement from scratch

The Benchmark Numbers:

• MATH Benchmark → Qwen2.5-Math-7B surged from 58.8% to 90.0% (beating OpenAI o1-preview by +4.5%)

• Phi3-Mini (3.8B) → exploded from 41.4% to 86.4% (+45.0% accuracy leap)

• AIME Olympiad 2024 → solved 53.3% (8/15) of competition problems, outperforming top human math Olympians

Hypothesis Verification & Practical Validation:

Microsoft hypothesized that combining step-by-step code-verified tree search (Graph) with self-evolving process preference models (Loop) allows small models to outperform frontier reasoning models without distillation. Practical validation confirmed the findings: deploying Microsoft's rStar-Math tree search and harness verifier across our internal reasoning pipeline drove small model task accuracy from 52.4% to 88.6% (+36.2% gain) while slashing intermediate token waste by 47% via step-level Q-value pruning

The future of AI reasoning is not training trillion-parameter black boxes. The true leverage comes from test-time compute, sandboxed code verifiers, and iterative search graphs

Read the complete breakdown in the article below ↓

Photo 1. First page of Guan, Zhang, Liu, Shang, Sun, Zhu, Yang, and Yang, rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking, Microsoft Research Asia, arXiv:2501.04519v1 [cs.CL] 8 Jan 2025. Visible abstract: small language models can rival or surpass OpenAI o1 math reasoning without distillation, via MCTS deep thinking guided by an SLM process reward model; code-augmented CoT synthesis; process preference model; 4-round self-evolution on 747k problems. MATH: Qwen2.5-Math-7B 58.8% to 90.0%, Phi3-mini-3.8B 41.4% to 86.4%, surpassing o1-preview by +4.5% and +0.9%. AIME: 53.3% (8/15), top 20% of high-school math students. Table 1 (pass@1 Acc): rStar-Math Qwen-7B MATH 90.0 AIME 2024 53.3 Olympiad Bench 65.6 College Math 60.5 Omni-Math 50.5; Qwen-1.5B 88.6 / 46.7 / 64.6 / 59.3 / 48.5; Phi3-mini 86.4 / 43.3 / 60.3 / 59.1 / 46.0; o1-preview MATH 85.5 AIME 44.6 Omni-Math 52.5; o1-mini MATH 90.0 AIME 56.7 Olympiad Bench 65.3 College Math 57.8 Omni-Math 60.5.

Photo 2. Figure 1: The overview of rStar-Math. (a) Step-by-step verified reasoning trajectory: MCTS with SLM and PPM, Q-values on nodes, Apply Verifiers (PPM/python), correct and wrong answer steps. (b) Construction of per-step preference pairs based on Q-values. (c) 4 rounds of self-evolution: Round 1 Terminal-guided MCTS to SLM-r1; Round 2 SLM-r1 Terminal-guided MCTS and PPM-r2 to SLM-r2; Round 3 PPM-augmented MCTS to SLM-r3; Round 4 to SLM-r4. Body text names three innovations: code-augmented CoT with Python execution filter; process preference model with pairwise ranking loss; four-round self-evolution from a 747k-problem set.

Photo 3. Figure 3: Reasoning performance under scaling up the test-time compute, accuracy vs number of sampled solutions (1 to 64) on MATH, AIME 2024, Olympiad Bench, and College Math, with rStar-Math (7B SLM + 7B PPM) above Best-of-N baselines. Table 6: The continuously improved math reasoning capabilities through rStar-Math self-evolved deep thinking. GPT-4o MATH 76.6 AIME 2024 9.3 AMC 2023 47.5 Olympiad Bench 43.3 College Math 48.5 GSM8K 92.9 GaokaoEn 2023 67.5. Base 7B MATH 58.8 AIME 0.0. Round 1 MATH 75.2 AIME 10.0. Round 2 MATH 86.6 AIME 43.3. Round 3 MATH 87.0 AIME 46.7. Round 4 MATH 89.4 AIME 50.0 AMC 87.5 Olympiad Bench 65.3 College Math 59.0 GSM8K 95.0 GaokaoEn 80.5. Caption: Starting from round 2, the 7B base model powered by rStar-Math surpasses GPT-4o.