saved
Training AI to Paint with Code
gist
Surya Narreddi and collaborators train a language model with reinforcement learning to paint watercolour hibiscus images by writing editable p5.brush JavaScript rather than emitting pixels. Absolute multi-judge rewards correlated and saturated, so the model plateaued on clip-art flowers. Pairwise comparison against a hand-rated reference pool, plus a short GEPA-evolved allowlist prompt, learned faster and produced terser winning code. The piece frames creative RL as designing reward structure so taste can generalize.
ideas
- Code is the editable artefact. Writing p5.brush sketches instead of pixels lets humans revise the image without returning to the model prompt.
- Aesthetic RL is a reward-design problem. Verifiable signals work for math and games; subjective quality needs carefully authored criteria or the model converges or drifts.
- Correlated judges waste gradient. Nine absolute scores measured the same quality repeatedly while length ramps saturated early, locking the model on flat clip-art flowers.
- Pairwise taste against a hand-rated pool. Judging rollouts against love-tier references opened dynamic range, beat the old plateau faster, and compressed winning sketches under 2,000 tokens.
- Short allowlists beat long API docs. GEPA evolved a prompt with eight brush methods and no documentation; long references caused confident, well-formatted API hallucinations.
quotes
“When you make an image with an AI model, the only way to participate is the prompt.”
“The code is the artefact, and the code is editable.”
“RL works when the reward is verifiable. A math problem is right or wrong.”
“Long reference documentation in a system prompt made the models hallucinate APIs.”