Show HN: Orchest – Data Science Pipelines

There's a large tradition of data analysis/BI tools using visual programming/ "no-code" tools, think Talend, etc. They're popular since data analysis typically contains a number of people without coding backgrounds. Even among data science teams, many don't have the engineering background of data engineers (and certainly there's a segment of data scientists who don't care as much about building the data pipelines and the cleaning that's necessary).

Low code tools _can_ (but not necessarily will) effectively empower data teams.

Thanks for your input. We are planning to also support an API and/or CLI (besides the GUI) to set up the pipelines through scripting. Depending on what we hear from the community we will definitely add this sooner rather than later.

You make a great point for the use of monitoring the progress of a run. During pipeline runs the visual pipeline interface (also in view only mode during or after running pipeline runs as part of an experiment) can be helpful as a visual indication of where the pipeline failed, how long pipeline steps took to execute and whether certain pipeline steps are still running.

Lastly, for team collaboration we think the visual interface is indeed helpful to explain (especially to the less technical people) the structure of the code, what exactly the pipeline sets out to solve and how the pipeline solves it. In addition, the possibility to add intermediary pipeline steps that serve as dashboards can be helpful for collaboration.

I agree, and so many data science and data engineering tools all seem to revolve around using notebooks, much to my frustration. I’ve worked in places whose data pipelines were built around seemingly infinite notebooks, all containing consistently poor software engineering.

It’s been enough to make me vow to not let people write notebooks that go into prod under my watch lol.

I’m constantly on the watch for software engineering focused tools that solve the issues, rather than data science/engineering focused tools. So many are inextricably linked into python as well, which doesn’t gel nicely with anywhere that has multiple languages in the codebase.

Currently, we are not focused on helping DS teams putting models into production.

It's those 'messy' projects with 10K LoC notebook that could easily be broken up into multiple steps (perhaps some of them notebooks, some of them Python scripts with library function usage) that we feel are a great match for pipelines in Orchest today.

When a team is still experimenting with what models to go with (i.e. trying neural networks v.s. decision trees) it can be helpful to have a more structured prototyping environment with reproducability and easier scalability. Which is also where Orchest shines.

If you want to test some of these models in production, you could easily push artificats to endpoints for serving, in the final steps of a pipeline.

Disclaimer: I've recently at my company built and released a product that has a displayed DAG on screen and allows people to build ML pipelines in addition to the existing data prep and blend capability. My company is profitable and the ML product is doing well. We also have the ability to have jupyter notebooks execute in-line with the DAG.

The comments on here are reflective of the fact that selling products like yours to typical coders is a dead-end. They don't like it or want it, not because it wouldn't be useful to them, but because they are attached to their current tooling and don't see a need for making their workflows more maintainable for non-coders. Some coders get it, after working in an office and realizing they are spending inordinate amounts of time tweaking their software for various data sources instead of focusing on the actual interesting stuff, but most don't.

Your product has the challenge of falling in a valley between the domain experts who can't code and therefore won't have notebooks available to pull into each node, and the people who can code and have notebooks not wanting to spend money or use a dirty, filthy, evil GUI instead of their precious glue code.

Your sweet spot of customers won't be on HN. They are the massive army of people with the job title "data scientist" who really aren't close to being actual data scientists. The enterprises are filled to the brim with them. Many of them can barely write Python or R, and are outright frauds. But plenty of them have a cursory knowledge of Python coding in Jupyter, and can actually do some Kaggle problems. However, basic ETL skills, web scraping, etc are foreign to them, and they have no ability to embed their shitty Jupyter notebook code into an ETL pipeline. These people aren't on HN. If they were, they wouldn't be so shitty at their job. Those people are your customers. The underqualified, borderline fraud data scientists who are EVERYWHERE. I sound like a cynical asshole.

But there's a silver lining here:

The best customers you can ever get aren't in that camp at all, and also would add a lot of value to their orgs using your tool. These folks are the domain expert analysts who really, really know their company's data and have the deep domain expertise. They are the people who data scientists have to talk to to be able to create value. You want to go after these people, and upskill them to basic python. People who know their business and it's data, and have just a little bit of data science skills create far more value than the opposite.

Scheduling periodic tasks can be useful for creating ETLs or to create alarms (every 1h hour check data for new values if condition is meet then send the email alert)

We want to try to make it easier for people to understand the landscape of tools and our position within it.

I personally like something like GitLab's https://about.gitlab.com/devops-tools/. We'll try to put up something similar on our website at some point.

I'm not deeply familiar with MLFlow, but from what I have seen/read it is more of a tracking framework that you can integrate into an existing codebase.

While Orchest allows you to take your existing codebase and structure it into a pipeline to get a visual and containerized way of interacting with the codebase (allow a mix of notebooks and .py/.sh/.R scripts), running the pipeline, and visually inspecting success/failure of pipeline runs/steps.

Another key point of difference is how we are more concerned with managing the flow of data. Since we let you build pipelines we can give you abstractions to separate data flow from the pipeline code. I.e. letting you define generic pipelines that take any data source (in some standard form, like a schema'd database) and produce reports. Because we control the data source in relation to the containerized pipelines we can also make sure the whole thing performs well when it's being executed in parallel (i.e. same version of the pipeline running grid search over paramaterized pipelines). In other words, we also control more of the underlying infrastructure when executing the pipelines.