The choice of groovy was unfortunate, but yet it still seems more popular than snakemake which I can only attribute to the nf-core set of curated workflows.
I have a dislike of nextflow because it submits 10s of thousands of separate jobs to our HPC scheduler which causes a number of issues, though they've now added support for array jobs which should hopefully solve that.
To implement an efficient dataflow-based programming API/DSL, you better have some support for channels and lightweight threads in a scriptable language, something that you've got in Groovy with the GPars library that Nextflow uses.
We opted for implementing all of this in Go in SciPipe, where we get similar basic dataflow/flow-based functionality as Nextflow with the native concurrency primitives of Go, but the Go syntax probably/surely puts away some biologists who have written some python at most before, and Go won't let us customize the API and hide away as much of the plumbing under nice syntax, as Groovy.
In this regard, Groovy with the GPars library for the concurrency, doesn't seem as a particularly bad choice. There weren't that many options at the time either.
The downside has been tooling support though, such as editor intelligence and debugging support, although parts of that is finally improving now with a NF language server.
Today, one could probably implement something similar with Python's asyncio and queues for the channel semantics, and there is even the Crystal language that has Go-like concurrency in a much more script-like language (see a comparison between Go and Crystal concurrency syntax at [1]), but Crystal would of course be an even more fringe langauge than Groovy.
Snakemake is easy to deal with that scenario. I had a profile for each of our slightly different hpc clusters. You could throttle the array by total resources so I could say request no more than 750gb memory allocated across the array to be polite to the rest of the hpc users, and it would fit however many jobs it could within that constraint and step of the pipeline. I could have a job instead be ran on the internet connected head node vs airgapped compute node if something needed downloading. Worked great and the python syntax is pretty useful along with conda env management baked in.
At a previous Biotech, we used Cromwell/WDL because the DSL was the most intuitive to our bioinformatics scientists. But seeing as that doesn't work as nicely on AWS (and is also supported by an organization that is imploding), we opted for Argo on our K8s cluster to process RNAseq data en masse. Getting the scientists to use YAMl has been an uphill struggle, but the same issues would apply to learning groovy I guess. We've found that the Argo engine is easier to maintain, and also we only have to support one orchestrator across our Bioinformatics and ML teams.
For industrial purposes, I've started to approach these pipelines as a special case of feature extraction and so I'm reusing our ML infrastructure as much as possible.
10 comments
[ 697 ms ] story [ 3299 ms ] threadNextflow and Snakemake are the two most-used options in bioinformatics these days, with WDL trailing those two.
I really wish Nextflow was based on Scala and not Groovy, but so it goes.
There is a Draft up for dsl3 that adds static types to the channels that I’m very excited about. https://github.com/nf-core/fetchngs/pull/309
I have a dislike of nextflow because it submits 10s of thousands of separate jobs to our HPC scheduler which causes a number of issues, though they've now added support for array jobs which should hopefully solve that.
We opted for implementing all of this in Go in SciPipe, where we get similar basic dataflow/flow-based functionality as Nextflow with the native concurrency primitives of Go, but the Go syntax probably/surely puts away some biologists who have written some python at most before, and Go won't let us customize the API and hide away as much of the plumbing under nice syntax, as Groovy.
In this regard, Groovy with the GPars library for the concurrency, doesn't seem as a particularly bad choice. There weren't that many options at the time either.
The downside has been tooling support though, such as editor intelligence and debugging support, although parts of that is finally improving now with a NF language server.
Today, one could probably implement something similar with Python's asyncio and queues for the channel semantics, and there is even the Crystal language that has Go-like concurrency in a much more script-like language (see a comparison between Go and Crystal concurrency syntax at [1]), but Crystal would of course be an even more fringe langauge than Groovy.
[1] https://livesys.se/posts/crystal-concurrency-easier-syntax-t...
For industrial purposes, I've started to approach these pipelines as a special case of feature extraction and so I'm reusing our ML infrastructure as much as possible.