HN Explorer

Show HN: I trained a 125M model to autocomplete piano on-device (simedw.com)

1 points by simedw ↗ HN
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15).

The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device.

The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.

117 comments

[ 0.99 ms ] story [ 28.7 ms ] thread
qmmmur
What is the goal here?

You have trained a model, presumably on other people's work, to improvise on an initial kernel of musical material. In doing so you:

1) remove the opportunity for one to develop their own skills in extending musical ideas 2) rip off other people's work wholesale 3) remove what is the most rewarding part of music (creating and shaping it through refinement and technique).

Razengan 

  What is the goal here?
To eventually prove that Intelligence™ and Creativity™ isn't special?

Or maybe there is some other "spark" deeper within?

When everyone can make anything as soon as they think of it, what will set us apart?

isoprophlex
man hits thing

thing go plink

machine hear plink

machine make many more plink

man happy for plink is fun

man not hit machine with club or scream on orange site

man leave cave and touch plant

perching_aix
4) it's fun and quaint, like an odd, jazzy version of a player piano from an alternate universe
evalystai
The idea is awesome! :) However there's definitely much room for improvement, first of all rythm and composition (so there's some sense of musical form).
isoprophlex
Amazing idea! Gonna hook this up to my little synthesizer and blast some square wave arpeggiated ML music!
heikkilevanto
I would love something like that, except that I play the melody, and it produces proper 3-4 part accompaniment, preferably in good baroque style. Extra bonus if it could also write it into a file in a format suitable for music editing programs.
vunderba
An early attempt at this was Microsoft Songsmith [1] all the way back in 2009, which would take a melody (usually recorded by mic) and try to scaffold an accompaniment around it though obviously not realtime in any sense of the word.

The closest we've had to realtime orchestration around a melody in the "real world" is probably arranger keyboards though your left hand is still responsible for the chord progression itself.

[1] - https://en.wikipedia.org/wiki/Microsoft_Research_Songsmith

Naitik88
This feels like a natural next step. Starting with a simple melody and having the system fill in the rest while still following your playing style could make it much more useful for experimentation than generating a complete piece from scratch.
altmanaltman
That is an incredibly hard challenge though. Creating the whole backing track (in any meaningful way other than just basic chords) from just melody will require an amazingly high number of highly subjective choices and random gen will not lead to good outcomes since our ears like intentional and artistical creativity in general.
zapperdulchen
Did you ever try Ludwig? It was rule-based by the Fritz chess engine maker. To my understanding Ludwig was grounded in traditional harmony, counterpoint, voice-leading, and orchestration principles taught in formal music education. It never took off so they eventually stopped it.

Some details: https://shop.chessbase.com/en/products/ludwig_3_engl

You can still download it for free: https://www.heise.de/download/product/ludwig-58854 (Heise is a renowned German publisher, nothing shady)

I am still waiting for something like this based on generaitive AI.

subhajeet2107
This is so amazing, can you improve the quality of generation at the cost of notes per seconds ? No one can play 108 notes/sec anyways, maybe you can train the model to do CoT for better quality
simedw
Yes, some kind of planning step is on my TODO list. Another thing I want to try is generating a few continuations in parallel, picking the one that looks best, and then continuing from there. Maybe the picking could be automatic.

I can probably squeeze out quite a bit more than 100 notes/sec as well. I haven’t spent much time optimizing inference yet.

devonsolomon
This is really fun. Scaler 3 starts with a chord progression and lets you break it down into musical performances and parts. Useful for ideation when producing.

Would be fun to get a midi clock going and play some chords on my piano and have my synth start jamming along with the bass and my keyboard doing some performance. Or any combination of the above.

mh-
+1 all of this. That would be incredible (this already seems very cool - excited to get home and try it!)
Naitik88
That would be a really interesting direction. At that point it starts feeling less like autocomplete and more like having another musician reacting to what you're playing in real time.
pil0u
Even after a few years deep into AI, I find your application absolutely magic. This is very inspiring, thank you for sharing.
leobg
I don’t have MIDI. How about whistling or playing the piano via microphone? Sounds easy. Another 6 month rabbit hole? :)
goda90
Reminds me of this project to generate every melody possible algorithmically in order to fight music copyright lawsuits. https://allthemusic.info/
ronyeh
Thanks. I enjoyed watching the TEDx talk on that page.
jwr
> Eventually I used Gemini 3.5 Flash for pairwise evaluation

But, but… wouldn't that be… (gasp) DISTILLATION?

Fun project!

davidajackson
How would you expand this to support elements like attack ("velocity of the key-down" in piano speak), grace notes, timing etc. Would each of those be part of this model or another model? How would you model an arbitrary element (pedal, duration, etc...)
arkmm
Very cool! Can you say a little bit about the size of the DPO training examples and how long training took?
simedw
For DPO I only had around 700 preference examples, so not much data at all. That took about 12 minutes to train on a single GPU.

Pretraining was obviously a a lot slower, the 125M model took roughly half a day.

hliyan
Ah, MIDI files. The only type of music you could realistically download from the internet back in the day, and you had to wake up at ungodly hours so that your dialup modem would not rack up a massive phone bill.
killerstorm
MIDI is still widely used for professional music production. It sounded goofy back in the day because synthesizer it was played on was not very good.
HPsquared
I'm thinking canyon.mid on Microsoft GS Wavetable Synth.
andai
I remember at one point RuneScape switched from the built in Microsoft midi whatever to their own sound engine, and from that day, everything sounded wrong, even the frogs, because to me, the crappy midi sounds were the whole personality and feel of the game.
otabdeveloper4
MIDI is a protocol, and it's not ever going away or being replaced.

Some protocols (like I2C or MIDI) are gonna be with humanity forever, probably.

dyzone
It's definitely possible to create amazing music with just soundfonts and midi files. Many DOOM wads ship with tons of beautiful music. https://www.youtube.com/watch?v=foaaoQVjXtk The song this video talks about uses default microsoft soundfont.
xtracto
I remember downloading MOD files. It was just a bit larger than midi, but sounded better. I had a PC but I think it was an Amiga thing
andai
Need For Madness is one of my favorite games, and probably two thirds of that fondness is the music, which was a bunch of MOD files.
gaya3bollineni
Running a 125M model on-device at that speed is impressive. How much did you have to optimize the model to get that performance on an iPhone?
simedw
The biggest speed improvement came from changing the note representation when I switched to compound note events: roughly 5× fewer autoregressive passes per note.

For the current model I’m using Core ML, which optimizes the kernels the first time you run it. I haven’t actually spent that much time tuning performance beyond that.

Naitik88
The answer about changing the note representation was interesting. Sometimes a change in how the problem is represented ends up giving a much bigger improvement than trying to optimize the model itself.
speedgoose
Talking about AI music with some live human MIDI inputs, Magenta Realtime 2 was released a few weeks ago and is pretty fun.

https://magenta.withgoogle.com/magenta-realtime-2

throw_m239339
I do use Google DDSP to generate trumpet and saxophone audio in real time from MIDI sequences. I must say it's one of the best acoustic instrument model so far and it's fast.

There are plenty of AI use in music that aren't about an LLM making music in place of the musician, like Suno & co...

the_black_hand
Cool work. I tried using LLMs to parse sheet music and they are really bad.
aghilmort
really incredible work! great use case, impeccable learning strategy, congrats!
[deleted]
(comment deleted)
mandeepj
Gemma 4 E2B was too heavy for your needs?
jasonjmcghee
I think this is a great project and very HN. Not sure why the comments are so focused on the deliverable- you learned way more and had a much more interesting experience.

One think I didn't see mentioned in the post- maybe I missed it- how large was the data? How many samples did you use to pretrain and post-train

munksbeer
There is this in the post:

> The final dataset contained a few hundred thousand MIDI files, representing roughly 300 million note events.

Tepix
Oh, a cool idea! I just tried it, works pretty well. Kudos!

One feature request:

Instead of playing the AI-generated audio solely through the iPhone's speakers, add an option to send the audio as midi notes to a device (probably the same one you received the mini notes from).