Show HN: Dstack – a command-line utility to provision infra for ML workflows (github.com)
Unlike traditional dev workflows, ML workflows are difficult to run on a local machine (due to the lack of memory, more CPUs/GPUs, etc). This is why people often have to use remote machines (e.g. via SSH), or adopt one of the end-to-end MLOps platforms.
Using remote machines is not difficult but it is tedious and requires a lot of manual actions. Using MLOps platforms on the other hand automates the manual work but requires the use of an opinionated interface, which often kills developer productivity.
Imagine, if you could run your ML workflows the very same way as you do it locally, but they would actually run in the cloud. And you wouldn’t need to worry about provisioning infrastructure, setting up the environment, etc.
I’m excited to show you dstack, an open-source tool that does exactly that.
It’s a command-line utility that allows you to run any workflows while it provisions infrastructure, setup the environment, and copies code/data for you. No need to install or configure anything in your environment or cloud. Simply install the CLI and run it.
The launch blog post: https://mlopsfluff.dstack.ai/p/simplifying-the-mlops-stack
We’d love to hear your thoughts and ideas. I’ll be here to answer any questions you might have.
8 comments
[ 3.7 ms ] story [ 38.0 ms ] threadURLs to 10M images.
I need to create a thumbnail for each of them.
I.e. I need to download each image, resize and save in the webp format to the S3.
Right now I am doing this locally and it takes forever, as my images are in high resolution their total size is a few Tb.
Is there a tutorial that will describe how to do it? I was thinking about using AWS Lambda + SQS, but if DStack is an easier solution, I would be happy to check it out.
The entire setup will take roughly 5-10 mins. Promise to add a tutorial with exact steps and code snippets soon.
In the future, we also plan to support distributed workflows. Actually, this is already supported by the design, we just need to implement a provider!
Here's the basic way of using them:
A workflow may produce output files to a local folder. In your workflow declaration, you can mark what folders to treat as output artifacts. Then, dstack would save output artifacts automatically, and you'll be able to reuse them via the unique name of the run, or a user tag assigned to this run.
All artifacts are stored in the S3 bucket that is configured for dstack. dstack is capable of syncing artifacts at start//end of the workflow or mount artifact folders via FUSE (of course if that is needed).
Each artifact is stored using the following path: <s3 bucket>/artifacts/<run name>/<job name>
A run can have multiple jobs, e.g. if it's a distributed workflow.
In future, we also think of providing a high-level Python API for accessing/storing artifacts.
Please share your thoughts and feedback!
As I said in the comment above, we think of providing a high-level Python API. THis API will allow to avoid copying the artifacts as an initial step.
But of course would love to hear what you think! ;-)