rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking

gist

rStar-Math from Microsoft Research Asia shows small models can rival or beat OpenAI o1 on math without distillation. A policy SLM searches at test time with Monte Carlo Tree Search, guided by an SLM process reward model. Training uses code-checked CoT from MCTS rollouts, a process preference model instead of naive step scores, and four self-evolution rounds on 747k problems. On MATH, Qwen2.5-Math-7B goes from 58.8% to 90.0% and Phi3-mini from 41.4% to 86.4%. AIME 2024 is 53.3%, or 8 of 15 problems.

ideas

• Small models can match o1 without a teacher. The claim is not distillation: a policy SLM and an SLM process reward model search at test time until math accuracy rivals frontier System 2 models.
• Python execution gates the tree. Code-augmented CoT keeps only intermediate steps whose Python runs, so a correct final answer cannot launder broken intermediates into training data.
• Prefer steps, do not score them naively. Q-values from MCTS rollouts are too noisy as step labels, so the process preference model learns pairwise ranking of good versus bad steps.
• Four rounds grow coverage from scratch. Policy and PPM start weak, then iterate over 747k problems until competition-level items enter the training set.
• The scoreboard is MATH and AIME, not an internal pipeline. Qwen2.5-Math-7B 58.8% to 90.0% and Phi3-mini 41.4% to 86.4% are paper results; AIME 53.3% (8/15) is top 20% of high-school olympiad students.

quotes