saved
Training a 125M-parameter Model to Autocomplete Piano
gist
Simon Edwardsson trains a 125M-parameter transformer to autocomplete piano MIDI in real time on an iPhone, about 108 notes per second. The useful work is representation and data: a single NOTE token carries pitch, onset delta, duration, and velocity so the backbone runs once per note, plus aggressive cleaning of a few hundred thousand piano MIDI files. Cross-entropy learns mechanics; DPO on Gemini pairwise continuation judgments made rollouts reliable. The result ships on-device as RollTab, still loop-prone on short prompts.
ideas
- Predict one complete note per step. A NOTE token bundles pitch, onset delta, duration, and velocity so the transformer backbone runs once per note, not once per field.
- Bake sustain into duration. Pedal-down extensions become sounding length, so the model never has to emit pedal events.
- Clean data beats more data. A few hundred thousand piano MIDI files, about 300 million notes, beat a five-times-larger noisy set.
- Cross-entropy learns mechanics, not taste. Music has many valid next notes, so next-token loss is a weak proxy for a good continuation.
- DPO on continuation preference made it reliable. Consensus Gemini pairwise judgments at β=0.03 beat the base model 69 percent of the time.
quotes
“Think GitHub Copilot, but for piano.”
“Instead of spending four transformer passes generating the attributes of a note, the transformer advances the music by one complete note at a time.”
“Cleaning and selecting the data mattered more than simply adding more of it.”
“DPO made the biggest difference after pretraining.”