What were some of the pain points you face(d) - looking back at your Metaflow adoption? Disclaimer: I work in Netflix ML Platform that helped open-source Metaflow originally.
Happy to help either through our gitter chat or help@metaflow.org.
Thanks for reporting it. We ll fix it. Sorry for the inconvenience.
Thanks. Let us know how you like the prototyping -> scaling out & up journey.
I wouldn't exactly say that. Jupyter notebooks don't have an easy way to represent an arbitrary DAG. The flow is more linear and narrative like. That said, we do expect metaflow (with client API) to play very well with…
Thanks for sharing the context. Hopefully we can have a (fast) follow up with Kube integration depending on demand.
Yes - you should be able to use dask the way you say. Your first part of the understanding matches my expectation too. Dask single box parallelism achieved by multi processing - akin to parallel map. And distributed…
I guess (?) - minus the input spec being not YAML but more language native (pythonic for e.g.)
Yes - that’s our thinking too. Compilers finding your typos for variable names seems helpful for user productivity.
Thanks for pinging on this. re: Kubeflow - imho it is quite coupled to Kubernetes. We don’t intend to be tied to a specific compute substrate even though the first launch is with AWS. We do follow a plugin architecture…
I would also add - dependency management (certain degree of reproducibility) as a first class feature leveraging conda.
Our hope with metaflow is to make the transition to production schedulers like Airflow (and perhaps similar technologies) seamless once you write the DAG via the FlowSpec. The user doesn’t have to care about the…
With many objects under the same S3 bucket - say for a flow or a run (with many tasks).
1. Metaflow should best help when there is an element of collaboration - so small to medium team of data scientists. Collaborating with your self is also another scenario when Metaflow can be useful since it takes care…
What were some of the pain points you face(d) - looking back at your Metaflow adoption? Disclaimer: I work in Netflix ML Platform that helped open-source Metaflow originally.
Happy to help either through our gitter chat or help@metaflow.org.
Thanks for reporting it. We ll fix it. Sorry for the inconvenience.
Thanks. Let us know how you like the prototyping -> scaling out & up journey.
I wouldn't exactly say that. Jupyter notebooks don't have an easy way to represent an arbitrary DAG. The flow is more linear and narrative like. That said, we do expect metaflow (with client API) to play very well with…
Thanks for sharing the context. Hopefully we can have a (fast) follow up with Kube integration depending on demand.
Yes - you should be able to use dask the way you say. Your first part of the understanding matches my expectation too. Dask single box parallelism achieved by multi processing - akin to parallel map. And distributed…
I guess (?) - minus the input spec being not YAML but more language native (pythonic for e.g.)
Yes - that’s our thinking too. Compilers finding your typos for variable names seems helpful for user productivity.
Thanks for pinging on this. re: Kubeflow - imho it is quite coupled to Kubernetes. We don’t intend to be tied to a specific compute substrate even though the first launch is with AWS. We do follow a plugin architecture…
I would also add - dependency management (certain degree of reproducibility) as a first class feature leveraging conda.
Our hope with metaflow is to make the transition to production schedulers like Airflow (and perhaps similar technologies) seamless once you write the DAG via the FlowSpec. The user doesn’t have to care about the…
With many objects under the same S3 bucket - say for a flow or a run (with many tasks).
1. Metaflow should best help when there is an element of collaboration - so small to medium team of data scientists. Collaborating with your self is also another scenario when Metaflow can be useful since it takes care…