I built writeup.ai to learn more about machine learning. I underestimated the amount of time it would take (thought 1 month, took 3, classic 3x off) ... alas, the feature creep of my judgement was hard to manage.
I learned WAY more about deploying ML in production environments than neural networks. ML DevOps was surprisingly hard. the ecosystem is a bit fragmented with managing TensorFlow / PyTorch / Docker / Nvidia dependencies. since there's SO MANY ongoing changes they have a lot of out-of-date documentation.
the backbone of the algorithms running this is from gpt-2 (openai's text generation algorithm). i experimented with almost all of the other text models including bert, transformer-xl, xlnet. i looked into ctrl (salesforce). most of them had incredibly high memory requirements, speed wasn't the fastest, or the output wasn't as good as gpt-2. this isn't a knock on the other algorithms, they were designed with the focus of answering questions in test sets (reading comprehension, etc).
i fine-tuned gpt-2 medium w/gradient checkpointing for five different datasets: legal, harry potter, game of thrones, song lyrics, academic research. training was relatively straight forward, but the data scrubbing was incredibly tedious. i'm still training/updating the research models.
app hosted on google cloud. everything is run via docker containers (which is also hard w/nvidia). autoscaling on usage. the main inference runs off from either cascade lake (cpu, surprising reasonable performance), k80s, or p100s. i experimented with the speed and profile of them a lot.
i'm going to write a detailed blog post about the code/deployment/infrastructure soon if you're interested. love to answer any questions and get feedback! thanks!
2 comments
[ 3.3 ms ] story [ 12.3 ms ] threadI learned WAY more about deploying ML in production environments than neural networks. ML DevOps was surprisingly hard. the ecosystem is a bit fragmented with managing TensorFlow / PyTorch / Docker / Nvidia dependencies. since there's SO MANY ongoing changes they have a lot of out-of-date documentation.
the backbone of the algorithms running this is from gpt-2 (openai's text generation algorithm). i experimented with almost all of the other text models including bert, transformer-xl, xlnet. i looked into ctrl (salesforce). most of them had incredibly high memory requirements, speed wasn't the fastest, or the output wasn't as good as gpt-2. this isn't a knock on the other algorithms, they were designed with the focus of answering questions in test sets (reading comprehension, etc).
i fine-tuned gpt-2 medium w/gradient checkpointing for five different datasets: legal, harry potter, game of thrones, song lyrics, academic research. training was relatively straight forward, but the data scrubbing was incredibly tedious. i'm still training/updating the research models.
reddit has made some amusing harry potter and game of thrones fanfiction w/this so far: https://www.reddit.com/r/FanFiction/comments/d5s9yh/i_made_a...
app hosted on google cloud. everything is run via docker containers (which is also hard w/nvidia). autoscaling on usage. the main inference runs off from either cascade lake (cpu, surprising reasonable performance), k80s, or p100s. i experimented with the speed and profile of them a lot.
open-sourced at https://github.com/jeffshek/writeup-frontend, https://github.com/jeffshek/open
i'm going to write a detailed blog post about the code/deployment/infrastructure soon if you're interested. love to answer any questions and get feedback! thanks!
Just have to pick your head up and continue.