Launch HN: Sematic (YC S22) – Open-source framework to build ML pipelines faster
I started my career searching for Supersymmetry and the Higgs boson on the Large Hadron Collider at CERN, then moved to industry. I spent the last four years building ML infrastructure at Cruise. In both academia and industry, I witnessed researchers, data scientists, and ML engineers spending an absurd share of their time building makeshift tooling, stitching up infrastructure, and battling obscure systems, instead of focusing on their core area of expertise: extracting insights and predictions from data.
This was painfully apparent at Cruise where the ML Platform team needed to grow linearly with the number of users to support and models to ship to the car. What should have just taken a click (e.g. retraining a model when world conditions change – COVID parklets, road construction sites, deployment to new cities) often required weeks of painstaking work. Existing tools for prototyping and productionizing ML/DS models did not enable developers to become autonomous and tackle new projects instead of babysitting current ones.
For example, a widely adopted tool such as Kubeflow Pipelines requires users to learn an obscure Python API, package and deploy their code and dependencies by hand, and does not offer exhaustive tracking and visualization of artifacts beyond simple metadata.
In order to become autonomous, users needed a dead-simple way to iterate seamlessly between local and cloud environments (change code, validate locally, run at scale in the cloud, repeat) and visualize objects (metrics, plots, datasets, configs) in a UI. Strong guarantees around dependency packaging, traceability of artifact lineage, and reproducibility would have to be provided out-of-the-box.
Sematic lets ML/DS developers build and run pipelines of arbitrary complexity with nothing more than minimalistic Python APIs. Business logic, dynamic pipeline graphs, configurations, resource requirements, etc. — all with only Python. We are bringing the lovable aspects of Jupyter Notebooks (iterative development, visualizations) to the actual pipeline.
How it works: Sematic resolves dynamic nested graphs of pipeline steps (simple Python functions) and intercepts all inputs and outputs of each step to type-check, serialize, version, and track them. Individual steps are orchestrated as Kubernetes jobs according to required resources (e.g. GPU, high-memory), and all tracking and visualization information is surfaced in a modern UI. Build assets (user code, third-party dependencies, drivers, static libraries) are packaged and shipped to remote workers at runtime, which enables a fast and seamless iterative development experience.
Sematic lets you achieve results much faster by not wasting time on packaging dependencies, foraging for output artifacts to visualize, investigating obscure failures in black-box container jobs, bookkeeping configurations, writing complex YAML templates to run multiple experiments, etc.
It can run on a local machine or be deployed to leverage cloud resources (e.g. GPUs, high-memory instances, map/reduce clusters, etc.) with minimal external dependencies: Python, PostgreSQL, and Kubernetes.
Sematic is open-source and free to use locally or self-hosted in your own cloud. We will provide a SaaS offering to enable access to cloud resources without the hassle of maintaining a cloud deployment. To get started, simply run `$ pip install sematic; sematic start`. Check us out at https://sematic.dev, star our Github repo, and join our Discord for updates, feature requests, and bug reports.
We would love to hear from everyone about your experience buil...
49 comments
[ 5.0 ms ] story [ 115 ms ] threadJoin our Discord in you want to discuss this further – https://discord.gg/4KZJ6kYVax
I wish it was as easy as drag and dropping the model to target servers after building the model.
I think ML development is where web development was in the late 2000's. Clear patterns and best practices have not yet emerged. For example, many ML developers work without reproducibility and traceability enabled. This does not allow for fast and safe work.
disclaimer: this is a nice framework, will happily try it.
Imho: The underlying patterns are quite clear, and there are various approaches to build stable pipelines.
I have used automation with basic containers + gitlab actions / custom runners, clearml, earthly pipelines, kubeflow,.. for this.
All of those can give reproducibility (experiment tracking, code & dependencies, etc.) without much effort.
The last mile (model deployment) is often very specific, so let's keep that out of scope.
But: The basic problem is cultural, not technical.
One stated goal of this project hits close to the root cause: "Facilitating the transition from Jupyter Notebook prototype code to steady production-grade pipelines".
As ML developers, we have to stop regarding notebooks as anything that produces acceptable output (apart from initial exploration). Work has to happen in structured, tracked, and versioned codebases (~production code).
Anything that happened locally/in a notebook might as well not exist from my point of view.
Of course we understand why Notebooks are so appealing and useful, and we think they have their place in the toolbox (like a Python console for developers).
And you are correct that there already are many tools to build reproducible traceable pipelines. What we have found is that they are still too difficult to adopt. Which is why we are trying to greatly lower the barrier to entry.
What a time to be alive.
All jokes aside, this is really awesome and i'm glad to see more and more tools to make ML more developer friendly and accessible. Out of curiosity do you guys come from a TF, pytorch, jax, etc background?
However, it's only required when running pipelines in the cloud. When running locally (`$ sematic start`), nothing else than Python is required.
Regarding background, we are folks with experience building ML tooling. We've built infra around TF and Pytorch, but none with Jax. That being said, Sematic is agnostic to the framework you want to use.
Kubernetes becomes interesting when using heterogeneous resources (e.g. GPU nodes for training, high-memory for data processing, etc.), but is not a necessity.
In the coming months, we will provide a fully-hosted SaaS offering to save you the hassle of maintaining your own infrastructure.
Disclaimer: author of Orchest https://github.com/orchest/orchest
Btw, orchest looks like a cool way to orchestrate notebooks!
Also, hi! I'm founding engineer here at Sematic. Happy to answer any questions I can!
Our approach has been supporting both Python scripts (R, Julia, JS and Bash too for that matter) and notebooks as it would give users the ability to choose the right tool for the job and in case of migrations from notebooks to scripts make the process more incremental.
Welcome to the thread :wave:
UI: whereas Metaflow provides the ability to build your own result visualizations explicitly in your workflow (via their "cards" feature), Sematic makes it so that your outputs (and inputs) get automatic rich visualizations based on the data type of the data being passed around.
API: Instead of being based around explicitly building up a graph, where you have to explicitly specify the I/O connections between steps, Sematic makes defining your steps look like writing/calling python functions.
Packaging: Whereas metaflow requires you to include packaging information in the code defining your steps (the @conda decorator, etc.), Sematic plugs into your existing dependency management to bundle up dependencies for execution in the cloud.
Edit: Digging deeper, Sematic relies on Bazel (https://docs.sematic.dev/execution-modes#dependency-packagin...) and needs a BUILD file to specify all the dependencies for cloud execution. It seems that the entire pipeline will execute as a single (or multiple) k8s pod(s) using the same environment?
I am quite interested in trying out Sematic. Any guidelines on what kind of scale Sematic can support today (and the near future)?
> It seems that the entire pipeline will execute as a single (or multiple) k8s pod(s) using the same environment?
single docker image, but multiple pods (when you are using the full cloud mode). This was an intentional decision to avoid confusion around what things could be imported in what places (mimicking more what it would be like in one python instance), and also avoid weird version inconsistencies across the pipeline.
> Any guidelines on what kind of scale Sematic can support today (and the near future)?
Based on some prior tooling experiences, the main bottleneck should be what your Kubernetes cluster can handle.
> I am quite interested in trying out Sematic
Glad to hear it! We'd love to hear about your experiences. You can join our discord if you want help while you're trying it out: https://discord.com/invite/4KZJ6kYVax
I think this is a great way to explain what you're doing. I'm working in the same space (ML/DS tooling) and I feel like we, as the ML/DS community, haven't cracked exactly how Rails for data looks like, I actually wrote some ideas on this a while ago (https://ploomber.io/blog/rails4ml/).
Congrats on the launch and best of luck with the product!
We think that is a good way to go: solid abstractions that experts can build on top of, but also more junior folks can get started quickly by following best practices.
Ploomber looks great too, I like the breakdown of your SaaS offering.
> Sematic makes I/O between steps in your pipelines as simple as passing an output of one python function as the input of another. Airflow provides APIs which can pass data between tasks, but involves some boilerplate around explicitly pushing/pulling data around, and coupling producers and consumers via named data keys.
In robotics you sometimes need high performance data transformation e.g. convert pile of raw robot log data protos --> pile of simulation inputs --> pile of extracted data --> munged into net input format
Does semantic support this if the communication between tasks uses python functions? Like if my simulator is C++, will I have to use SWIG?
In some of the competing systems, the input/output between nodes are just produced files as side effects, which is nice because it doesn't care what language / infra you use as long as you produce the required input/output.
What we have seen done in the past is use things like pybind11 to expose C++ APIs in Python, which I guess is a similar concept to SWIG. If you are using build tools such as Bazel, you can even get the C++ compiled at run-time when submitting your pipeline.
Regarding i/o artifacts, Sematic lets user choose how they are serialized. We offer reasonable baseline defaults, but certain artifacts require serialization formats that are cross-language (e.g. ROS messages since you mention robotics).
As a last resort, users are free to serialize and persist artifacts by hand as part of their pipeline functions (e.g. storing in a cloud bucket) and only returning a reference to said artifact (e.g. a Python dataclass with artifact location and metadata).
Your project is cool and has a cool name!
Both our projects are ML-adjacent. I don't want to cause any confusion, and will start thinking of some alternate names for my project.
Woof, I'm out.
A software engineer can easily use a XGB model and get some outputs, but is a long way from having a deep understanding of statistical distributions. A ML engineer can easily put together a Django website, but is far from designing a maintainable software application.
Here are some high level questions:
- How does it handle failure of individual tasks in the pipeline? - What if the underlying jobs (e.g. training or dataset extraction or metrics evaluation) need to run outside the k8s cluster (e.g. running bare-metal, slurm, sagemaker, or even a separate k8s cluster)? - How does caching work if multiple pipeline can share some common components (e.g. dataset extraction)?
> - What if the underlying jobs need to run outside the k8s cluster? You are free to launch jobs on third-party platforms from one of your pipeline steps. This is a pretty common pattern, for instance launching a Spark job, or a training job on a dedicated GPU cluster. In this case, the pipeline step that launches the job (the Sematic function) needs to wait for the third-party job to complete, or pass a reference to the job to a downstream step that will do the waiting.
> - How does caching work? At this time there is no caching (as mentioned Sematic is very new :). We will implement memoization soon. What you can do is run a data processing pipeline separately and then use the generated dataset as input to other pipelines. This is a pretty common pattern: having a number of sub-pipelines (e.g. a data processing loop, a train/eval loop, a testing/metrics loop, etc.) that you can run independently, but also you can put them together in an end-to-end pipeline for automation. Sematic lets your nest pipelines in arbitrary ways, and each sub-pipeline can still have its own entry-point for independent execution.
Do I have the option to access the underlying infra through a Unix shell, when the UI isn’t enough?
When running on cloud instances, Sematic runs container images on your Kubernetes nodes. You are free to SSH into those and run the images manually to have the same dependency context, but any ephemeral context of the job (e.g. storage) will no longer be there.
What does bother me here, much as with many other ML orchestration frameworks, is that examples and even docs don't tell me how you play with serious situations in terms of scale.
In fact, what you describe as "a real ML pipeline" is - in my view - not a good example of a real ML pipeline because, for instance, it doesn't tell me how I'd solve the issue of scaling out multi-node training when the data doesn't fit in a neat and standard PyTorch Map Dataset that loads some csv from the web.
I mean, maybe it's because I am dumb, and maybe the majority of people do in fact train single-node models on MNIST data, but I'd appreciate some more information on how you deal with more diverse sources in your pipeline. Will I have to squeeze cloud provider X's data solutions (which I am obligated to use by the client, say) into submission until they fit your examples? Because these days I get the feeling that claims of "easily orchestrating your ML pipeline" often amount to that. I see you started some of these topics in the "integrations" part of the docs. However, these pages do not seem to exist yet (for me). Furthermore, the "roadmap" link goes to a 404.
For me, these are the important topics. I can get a nice simple ML pipeline easily on Azure, AWS or Databricks if I am willing to conform to whatever they are doing already. It seems you are in a position to tackle more challenging problems, so that would be nice to show.
Cool product, and good luck!
We are a very new framework (launched publicly July 1st :-), so there is much work to be done to cover many more example use cases.
What we have found powerful about this plain function approach is that users can submit jobs on remote platforms (e.g. Spark, Google Dataflow, etc.), and use heterogenous resources (e.g. standard nodes to launch third-party jobs, then GPU nodes for training, etc.). So whatever "cloud provider X's data solutions" you have to use, if it has a Python API to submit and wait for jobs, you should be fine.