Just curious if you looked at Apache Airflow as the integration functionality and pipeline / DAG configuration approaches are similar and it's also done in Python.
It's a relatively simple Nim code. It takes a line that starts with "import" and searches for three components: module, function and alias. If you don't provide an alias, you'll get an index error.
Should have named it Pypelines. But seriously though I am not sure what the benefits over multiprocessing are. It looks a little cryptic at first sight.
17 comments
[ 2.8 ms ] story [ 55.5 ms ] threadJust curious if you looked at Apache Airflow as the integration functionality and pipeline / DAG configuration approaches are similar and it's also done in Python.
https://github.com/apache/airflow
The Nim language is new to me. What was the experience like implementing a framework in it?
You can trivially define a worker pool or arbitrary size and pipe a stream of values through your function using multiple cores, avoiding GIL issues.
If you have something which scales to multiple machines just use celery or redis streams.
The code splits the line in function names, args and those operators, to then orchestrate the pipeline flow.
Small bug in the example code? I think this:
Should be this: