Can AI propose interesting research questions?

gist

After42, a Stanford-led group, built an agent workflow that develops research questions by trying to solve them, then had experts grade them. Across math, statistics, theoretical computer science and physics, 29 experts judged 45 of 135 final questions interesting: 33 of 70 in phase transitions in random structures, none of 10 in plasma physics. Model judges' shortlists match the experts' consensus picks about as well as individual experts do, but miss questions that only one expert loved.

ideas

• Proposing alone yields mostly duds. Asked straight out, a literature-grounded model writes questions that pass novelty (90%) and difficulty (77%) checks, yet experts call only 3% interesting. Filtering plus rounds of solve, refine and critique lifted that to about a third in the phase-transitions study.
• Solving reshapes the question. In the worked example, solution attempts on a random geometric graph found extra triangles, so the question became whether the graph is Erdős–Rényi plus a Poisson number of triangles. Eight of nine experts ranked it top 15; the model ensemble ranked it 36th.
• Most questions are recombinations. 97 of 100 initial proposals and 61 of 70 final ones build directly on known work (A+B, A−B or the next step). Weak ones read as forced combinations, engineered complexity, or lemmas posing as problems.
• Model judges are mode seekers. At top 20, the four-model ensemble recovers 80% of the expert consensus set, but only 35–85% of any single expert's top 20, and does no better than chance on picks only one expert made.
• Past feedback personalizes better than stated taste. Giving the judge an expert's written preferences did not help; their feedback on 33 earlier questions raised one expert's rank correlation from 0.26 to 0.42.

quotes