> … schedule hundreds of thousands of workflows, millions of jobs every day and operate with a strict SLO of less than 1 minute of scheduler introduced delay even when there are spikes in the traffic.
I wouldn’t necessarily call it over engineered if it can handle this type of workload.
But if you are running a fraction of these workloads with a much more relaxed SLO (service level objection), then yea this is overengineered.
Like any solution from FAANG, or “big tech”. It works for _them_ at scale but for 90% of companies the sheer management of this service will quickly outweigh the benefits.
> Like any solution from FAANG, or “big tech”. It works for _them_ at scale but for 90% of companies the sheer management of this service will quickly outweigh the benefits.
In other words, learn it now and get it on your resume, because within a year every 200-hit-per-day service and site is going to rearchitected to use it.
Here's a video of Primeagen, a YouTuber who uses to be a dev at Netflix, watching that video and then telling a very similar story that happened at Netflix: https://youtu.be/s-vJcOfrvi0
Not sure about the others but java is just an issue, if the devs did not care about cross platform and is mostly easy to fix if native libraries are absent.
I was just thinking didn't Netflix already have its own workflow engine: https://github.com/Netflix/conductor, but found out it's archived/EOL so this makes more sense now.
1) Scheduling seems to be primitive? (strategy: sequential)
2) That's seems to be just a DAG executor? No high-level frontend?
3) No executiun context?
4) No typings?
5) No concept of a stream?
It would be way too ambitious to call this thing an orchestrator, it seems to be just a primitive distributed DAG executor.
The link was changed at some point, from the original Medium article:
> Unlike traditional workflow orchestrators that only support Directed Acyclic Graphs (DAGs), Maestro supports both acyclic and cyclic workflows and also includes multiple reusable patterns, including foreach loops, subworkflow, and conditional branch, etc.
41 comments
[ 2.7 ms ] story [ 95.5 ms ] threadTo me seems to overengineered for most of the companies outside there, how many people do you need to manage it?
> … schedule hundreds of thousands of workflows, millions of jobs every day and operate with a strict SLO of less than 1 minute of scheduler introduced delay even when there are spikes in the traffic.
I wouldn’t necessarily call it over engineered if it can handle this type of workload.
But if you are running a fraction of these workloads with a much more relaxed SLO (service level objection), then yea this is overengineered.
Like any solution from FAANG, or “big tech”. It works for _them_ at scale but for 90% of companies the sheer management of this service will quickly outweigh the benefits.
In other words, learn it now and get it on your resume, because within a year every 200-hit-per-day service and site is going to rearchitected to use it.
Here's their tech blog post about it: https://netflixtechblog.com/orchestrating-data-ml-workflows-...
Takes a few mins to get all the dependencies and build, but it definitely works.
Perhaps you're missing some dependency steps before hand if you're hand cranking gradle.
This is not it however.
More discussion on official post: https://news.ycombinator.com/item?id=41037745
It would be way too ambitious to call this thing an orchestrator, it seems to be just a primitive distributed DAG executor.
It's hard to take your comment seriously when this was one of the earliest talking points.
> Unlike traditional workflow orchestrators that only support Directed Acyclic Graphs (DAGs), Maestro supports both acyclic and cyclic workflows and also includes multiple reusable patterns, including foreach loops, subworkflow, and conditional branch, etc.
https://netflixtechblog.com/maestro-netflixs-workflow-orches...