CEO here. Happy to make introductions to the team for anyone who is hiring. We have valuable residual knowledge about building a product in the MLOps platform space and achieving product-market fit, and I hope that this might be helpful to someone else.
I cannot imagine shutting down a company with a team. Must be really hard. Your note was refreshingly succinct, with genuine thank yous to the people that helped.
I don't have any advice or anything, just wanted to let you know that anyone that has ever tried something and failed understands how painful it is, and everyone else won't. Best of luck to you and your team as you all move forward.
We priced too low at first, tried too high when we released our enterprise product, did some compares with other dev tooling in sales convos with early customers and then settled on something in the middle ($600/user/month) and stuck with that cuz it seemed to be working. If we'd been around for longer we would have done more pricing tests but getting the right features to unlock larger deals (more users) was higher priority at the end. Btw, the delivery model that had PMF was customer hosts it in their own cloud, we ship terraform
Thanks again for the info. This is what I thought about self-hosting.
However, did customers raise concerns about hosting your code in their on perm clusters, For example, did they insist that you open source?
Also, when you host the code on the customer cluster, how did you track usage? For example, was there a process within the customer site, that would send usage information to your site ?
And, I would also imagine that every engagement is unique (not sure if this is correct),
Hence, did you outsource the solutions engineering or that was done as part of the team?
Hi, one of the engineers who created dotscience here. Customers didn't insist on open sourcing anything, we did have some open source components but I don't think they cared about that :) Regarding tracking usage - usually it was just many conversations with customers and having them on our Slack channel. Solutions engineering - mostly development team would be helping with writing anything specific that they need.
On the SaaS side we had a lot more analytics, used segment, intercom and internal "audit events" to better understand what's happening.
1. Deploying models easily (data scientist doesn't need to grok docker/kube)
2. Monitoring models including data and model drift/statistical monitoring (data scientists don't need to grok prom/grafana)
3. Only once these base concerns are met, model inventory, provenance, data versioning, reproducibility, collaboration with notebooks, ci integration etc.
Happy to talk more - drop me a note at luke@dotscience.com
Certainly have them check out the WiMLDS job board for positions that may fit their expertise, or contact SigOpt since they seem to be in the same space. I'm not affiliated with either, but know some good folks from both.
Sorry to hear this! Can you share what went wrong ? I’m really interested in the “DevOps for ML” space and I would love to hear what you think was the greatest difficulty or problem with your approach (if you are willing to share that).
I'd love to, genuinely, but I don't think it will be possible. https://github.com/dotmesh-io/dotmesh is already open source though which is one key component of Dotscience.
If AI/ML was just part of R&D then it's likely to get downsized or cut as companies refocus on their core products. If AI/ML is or is meant to be powering the core products or services, eg at banks, retailers, and manufacturers, then those teams are actually more important than ever. Because ML usually reduces labor and overhead, so the faster they can get it into production the faster they'll lower their operating expenses.
The challenge for dotscience was that there are so many players in this space that it's hard to stand out and win deals. See: https://zdnet3.cbsistatic.com/hub/i/r/2019/07/17/b17497a0-84... (These landscape charts are always made to look complicated on purpose, but the point is clear.)
> Thanks for checking out dotscience, unfortunately due to the current situation we have had to close our doors and are no longer operating. You may however enjoy checking out the MLOps community to find out more about all the tooling and best practices happening right now in this ever changing landscape.
> Here are some relevant links to the MLOps community: Slack, Youtube, Weekly meetups.
I got the pop-up but only on the first visit to the home page. Subsequent visits or a refresh do not trigger the pop-up. Firefox on Win. Noscript set to temporarily allow all.
It's really revealing that such amazing companies on paper...just don't make money at all and have to resort constantly to outside money to stay alive.
One thing I learned from working with startups in the space (and other B2B verticals): There is little correlation between how nice the site looks and how well the company is doing as a business. In the consumer space: maybe. But not in enterprise software.
Your comment is worded in a slightly mean way, but I don't disagree. I work on autonomous 3D navigation. Clicking on their "Autonomous Vehicles" solution I see:
- model management solutions
- enormous quantities of data
Well, I have docker containers for my data, and docker containers for the underlying TensorFlow versions, and docker containers for model source code that went into large-scale training. All of that is coordinated using a git repo which has some shell scripts to execute the correct model with matching data on a compatible TF image. So if someone says "I need you to re-run last week's model", I check out the script from that time and run it on an empty server.
I honestly don't get what MLOps is or why I would need it.
How many other ML people do you work with? Do they all use the same system? Are you only working with Tensorflow and no other tools?
Generally things like MLOps are targeted at providing systems for people that either don't know how, or don't have the time, to set something up themselves, especially when the data science team is more than just one or two people, and when they're using a range of tools that need to interoperate in various ways.
Something like Kubeflow, for example, does a lot more than what you describe.
2 others, and yes, everyone has access to the central git repo and to our shared private Docker repository.
And no, we use TensorFlow and Chainer, but both are python frameworks. Plus Numpy, of course.
I like that KubeFlow has an introduction video, but I find it quite odd, too. They talk a lot about how they will make things simpler, but then I learn that I'll need to run Kubernetes on my laptop, my servers, and potentially the cloud.
Plus they use irritating marketing phrases like "you can just focus on your model" or "let KubeFlow handle the abstraction of running on X". Most deep learning models nowadays are memory-limited, so improving the model may well mean optimizing GPU memory usage. And after trying to port a working model from Ubuntu + 1080 TI to Google Cloud + V100 and/or Google Cloud TPU, I distrust anyone who would treat those significant hardware differences as "just an abstraction".
So at the very least, I'll need to enforce consistent GPUs and Operating Systems among all servers, just to make things run OK everywhere.
I think a service that rents out GPU-equipped bare metal Ubuntu + Docker servers by the hour would be what I'd want to use.
If they also offer a private Docker repo, fast S3-compatible storage and some pre-built images with Jupyter preinstalled, that might be the entire ML pipeline that I need.
I would say that you don't need it if you don't feel hindered in your work. But it is solving a real need. Problems arise when you start to have larger teams who interact with models, when the models are applied in production at a regular cadence.
Having your model in docker generated from git is a good first step, but does not solve the most important issues when you are at scale: reproducible training, tracking of experiments including data, ML-specific observability for your models in prod, etc. See e.g. https://martinfowler.com/articles/cd4ml.html.
More concretely, since the above easily sounds like a buzzword soup:
1. Experiment-wise, you don't want just want to track your ML model definition, but also track the data and everything else used to build it, not just use it. A typical thing I have seen at every company I worked at: we have this model but we don't know how to reproduce it because the lost the data, or there was some magic numbers in training that may be on an internal wiki if you're lucky.
2. For some important use cases, you want to iteratively work on improving the model in production. That often means work on the data side, feature engineering, tracking skew prod vs training, etc. the model is not often changed. In almost every case, a useful model is a model that sees 100s if not more iterations in production. You need 1. to do 2.
3. Some of those tools are useful to enforce invariant or detect data issues, which is again very common when you run models for a long time. See e.g. tfdev.
But to go back to your point: MLOps is a 2nd order kind of thing. The first order is of course that most ML-related projects are useless, poorly conceived, or even lack any kind of quantitative analysis on the business and/or product. Companies should invest there before MLops IMO.
It is tough to make money, unless you have a solid budget for sales - even more so when you need sales people with some degree of competence in the field you're working in, and the product you're selling.
I think it's just took us too long to write the product as the scope was really big:
- CLI/UI.
- Main backend for storing projects, user accounts, managing pull requests, forks, runners, deployers, loadbalancers.
- Data backend (dotmesh).
- Auto provisioning of VMs with jupyterlabs running and data synced to GCP, AWS
- Runners that configure environment, install dependencies open up tunnels so users can access them and start working.
- Optimized machine imagine builds so the startup takes ~1min (some of the docker images like jupyter lab are very big)
- Model packaging into docker images.
- Model metrics capturing (a proxy that runs as a sidecar and intercepts requests) and then attaching relevant classes for your models.
- Kubernetes operator to deploy the actual models. User didn't have to worry about creating deployment manifests, services or ingresses (they wouldn't even care about docker images). They would just say which model to deploy and they would get a URL. Models could be deployed in a k8s cluster built from nodes with spot instances so would run pretty cheap :)
- Last component that I worked on was probably one of the most fun - an inference router that could allow canary deployments for models and also shadow deployments where traffic is sent to many models at once but responses are taken only from primary. We got a really nice UI for this as well where you could drag sliders around to configure % of traffic and so on. Unfortunately never managed to write docs for this.
- Terraform to wrap everything and deploy to GCP/AWS.
Our team was always quite small so we were stretched thin. In the end sales were going well as well, probably 6 more months and we would have broken even and then profitable :)
Btw https://mlops.community (which we started) will live on, Demetrios, Chris and Dan are kindly keeping it running ️
If you're interested in MLOps, come and join our weekly 9am PT / 5pm UK Wednesday online meetups - it's a great community and it's growing quickly - and now it's vendor neutral too :)
Interested in using it. I've kept an eye on dotscience and dotmesh for quite some time. At work, we've only very recently ramped up Data Science and ML to a point where tools like these would be really helpful.
I loved all the demos of dotscience I've seen at conferences. I'm saddened that we'll never get to try it, but glad to see that dotmesh will live on!
45 comments
[ 6.6 ms ] story [ 106 ms ] threadI don't have any advice or anything, just wanted to let you know that anyone that has ever tried something and failed understands how painful it is, and everyone else won't. Best of luck to you and your team as you all move forward.
So I am about to introduce a new product into this space, can you share how did you get your first 1-50 customers?
Also, Can you share general details about your customer demographic? I.e. what verticals, company size, etc?
Another question: can you elaborate more on how did you price the product? Did you do A/B testing on the price? ask the first customers?
However, did customers raise concerns about hosting your code in their on perm clusters, For example, did they insist that you open source?
Also, when you host the code on the customer cluster, how did you track usage? For example, was there a process within the customer site, that would send usage information to your site ?
And, I would also imagine that every engagement is unique (not sure if this is correct),
Hence, did you outsource the solutions engineering or that was done as part of the team?
So to sum up, what did customers care about the most from your point of view? (maybe top 3)
2. Monitoring models including data and model drift/statistical monitoring (data scientists don't need to grok prom/grafana)
3. Only once these base concerns are met, model inventory, provenance, data versioning, reproducibility, collaboration with notebooks, ci integration etc.
Happy to talk more - drop me a note at luke@dotscience.com
Leave a legacy, maybe the community will keep it going!
The challenge for dotscience was that there are so many players in this space that it's hard to stand out and win deals. See: https://zdnet3.cbsistatic.com/hub/i/r/2019/07/17/b17497a0-84... (These landscape charts are always made to look complicated on purpose, but the point is clear.)
It’s a good and vibrant community, and I’m learning a lot from the webinars as a practitioner. It’d be sad to see if it goes as well
What do you mean by this? I don't see any indication of them shutting down on the homepage, only on the blog.
> Dotscience
> Thanks for checking out dotscience, unfortunately due to the current situation we have had to close our doors and are no longer operating. You may however enjoy checking out the MLOps community to find out more about all the tooling and best practices happening right now in this ever changing landscape.
> Here are some relevant links to the MLOps community: Slack, Youtube, Weekly meetups.
- model management solutions
- enormous quantities of data
Well, I have docker containers for my data, and docker containers for the underlying TensorFlow versions, and docker containers for model source code that went into large-scale training. All of that is coordinated using a git repo which has some shell scripts to execute the correct model with matching data on a compatible TF image. So if someone says "I need you to re-run last week's model", I check out the script from that time and run it on an empty server.
I honestly don't get what MLOps is or why I would need it.
Generally things like MLOps are targeted at providing systems for people that either don't know how, or don't have the time, to set something up themselves, especially when the data science team is more than just one or two people, and when they're using a range of tools that need to interoperate in various ways.
Something like Kubeflow, for example, does a lot more than what you describe.
And no, we use TensorFlow and Chainer, but both are python frameworks. Plus Numpy, of course.
I like that KubeFlow has an introduction video, but I find it quite odd, too. They talk a lot about how they will make things simpler, but then I learn that I'll need to run Kubernetes on my laptop, my servers, and potentially the cloud.
Plus they use irritating marketing phrases like "you can just focus on your model" or "let KubeFlow handle the abstraction of running on X". Most deep learning models nowadays are memory-limited, so improving the model may well mean optimizing GPU memory usage. And after trying to port a working model from Ubuntu + 1080 TI to Google Cloud + V100 and/or Google Cloud TPU, I distrust anyone who would treat those significant hardware differences as "just an abstraction".
So at the very least, I'll need to enforce consistent GPUs and Operating Systems among all servers, just to make things run OK everywhere.
What I think might work is auto ml combined with models ops, or rather auto model ops.
Or even better: auto data managmenet -> auto pre-processes -> auto ml -> auto ops.
If they also offer a private Docker repo, fast S3-compatible storage and some pre-built images with Jupyter preinstalled, that might be the entire ML pipeline that I need.
http://papers.nips.cc/paper/5656-hidden-technical-debt-in-ma...
I use MLFlow at my company for solving two specific problems: tracking experiments and model performance.
But if you're small enough, you probably don't need MLOps, Devops, etc...
Having your model in docker generated from git is a good first step, but does not solve the most important issues when you are at scale: reproducible training, tracking of experiments including data, ML-specific observability for your models in prod, etc. See e.g. https://martinfowler.com/articles/cd4ml.html.
More concretely, since the above easily sounds like a buzzword soup:
1. Experiment-wise, you don't want just want to track your ML model definition, but also track the data and everything else used to build it, not just use it. A typical thing I have seen at every company I worked at: we have this model but we don't know how to reproduce it because the lost the data, or there was some magic numbers in training that may be on an internal wiki if you're lucky.
2. For some important use cases, you want to iteratively work on improving the model in production. That often means work on the data side, feature engineering, tracking skew prod vs training, etc. the model is not often changed. In almost every case, a useful model is a model that sees 100s if not more iterations in production. You need 1. to do 2.
3. Some of those tools are useful to enforce invariant or detect data issues, which is again very common when you run models for a long time. See e.g. tfdev.
But to go back to your point: MLOps is a 2nd order kind of thing. The first order is of course that most ML-related projects are useless, poorly conceived, or even lack any kind of quantitative analysis on the business and/or product. Companies should invest there before MLops IMO.
- CLI/UI.
- Main backend for storing projects, user accounts, managing pull requests, forks, runners, deployers, loadbalancers.
- Data backend (dotmesh).
- Auto provisioning of VMs with jupyterlabs running and data synced to GCP, AWS
- Runners that configure environment, install dependencies open up tunnels so users can access them and start working.
- Optimized machine imagine builds so the startup takes ~1min (some of the docker images like jupyter lab are very big)
- Model packaging into docker images.
- Model metrics capturing (a proxy that runs as a sidecar and intercepts requests) and then attaching relevant classes for your models.
- Kubernetes operator to deploy the actual models. User didn't have to worry about creating deployment manifests, services or ingresses (they wouldn't even care about docker images). They would just say which model to deploy and they would get a URL. Models could be deployed in a k8s cluster built from nodes with spot instances so would run pretty cheap :)
- Last component that I worked on was probably one of the most fun - an inference router that could allow canary deployments for models and also shadow deployments where traffic is sent to many models at once but responses are taken only from primary. We got a really nice UI for this as well where you could drag sliders around to configure % of traffic and so on. Unfortunately never managed to write docs for this.
- Terraform to wrap everything and deploy to GCP/AWS.
Our team was always quite small so we were stretched thin. In the end sales were going well as well, probably 6 more months and we would have broken even and then profitable :)
If you're interested in MLOps, come and join our weekly 9am PT / 5pm UK Wednesday online meetups - it's a great community and it's growing quickly - and now it's vendor neutral too :)
How will this affect dotmesh?
I loved all the demos of dotscience I've seen at conferences. I'm saddened that we'll never get to try it, but glad to see that dotmesh will live on!
I also filed a similar bug in github regarding open sourcing dothub.