One of the big lessons from advice like this: your organization must invest heavily in data infrastructure and unique SRE needs for machine learning if you hope to get value from ML projects.
If you don’t invest in data engineering or you yoke your ML engineers with job duties to also handle all the operations work, you’re not going to see meaningful returns and it won’t be because ML is just hype or is inapplicable to your business. It will because of poor leadership and overloading operations responsibilities.
For ML, the SDLC is just very different.
You need there to exist an auditable system of feature ingestion and serving as a foundation so that model training can standardize, and you don’t overwhelm an ML team with duties to constantly deal with schema changes, data ingestion job delays or outages, collating poorly maintained data across multiple teams.
You also need a standardized way to define training jobs, with easy config interface to specify all the resources needed (GPUs, disk space, cached data sets, certain CPU properties, docker container with the training environment prepared) that connects to an experiment management system to track models and map the entire bundle of parameters (hyperparameters, version control commit hash, resource settings, etc) to each trained model output artifact and evaluation metrics.
If you leave this stuff as free for alls for ML engineers to figure out, they will have to spend so much time fighting with infrastructure that is not their problem that either they’ll burn out and quit to get a job that doesn’t treat them this way, or they’ll resign themselves to being an “ops monkey” and you’ll see an endless churn of infra projects that are supposed to finally unlock the real potential of ML for your business but inevitably never do.
Practices of healthy companies that lead to real value from ML projects:
1. Make data producers responsible for their data delivery, data quality and data timeliness. If some team owns a search page, they also own everything about ingesting the data generated by that page. Data Platform teams might give them tools or provide infrastructure for it, but that application-specific team is ultimately responsible.
2. SRE teams do not helicopter in with parochial best practice recommendations and then leave. SRE teams must co-own the challenging components where the rubber meets the road and really be down in the implementation weeds - this is critically true for unusual or long-tail special case systems, for example like GPU workflows or Spark infra for ML jobs. SRE cannot be either siloed to focus only on the biggest use cases or acting like they just dole out philosophy and best practices. They have to be tied to the delivery incentives of every team they serve.
3. Clear the decks of ML engineer time. Your ability to get value from ML engineers directly relates to how much unblocked autonomy they are given. If they are overloaded with bureaucratic chores or maintenance tasks or on-call responsibilities, it is just directly subtracting from their comparative advantage of conducting model training and optimization. Nobody likes to admit it, but you can get other people to do the maintenance, the on-call shifts or the compliance tasks. Many engineers won’t be sacrificing a comparative advantage by being allocated to that stuff, but if you allocate ML engineers to it, you are flushing money down the toilet by wasting their comparative advantage. Remember: it’s a business - it’s not about what’s “fair” in terms of everyone sharing eg an on-call shift, it matters what creates the best result for your customer.
Before you can apply the tactical steps like the 12 steps of this article, you have to resolve much bigger issues of engineering culture to reflect the 3 points above.
If internal engineering politics won’t block you from these culture changes, then you can set things up to get value from ML. Otherwise, you’ll probably just waste money and have high turnover on the ML teams beca...
I wish I could copy and paste this on every low-effort/high-level "X tips for ML in production" article (for context, TFA's site was spamming Reddit and HN a while back). Most corporate decision makers don't know where to start when it comes to creating ML teams, so all this 12-factor stuff (PEP8 and SOLID, seriously?) isn't going to matter one wit if the necessary cultural and organizational changes aren't put in place first.
I see your point. To me, the two points however don’t stand at odds, both need to be addressed just the same. I said something similar in other comments, but only if your organization is building a strong team structure AND a solid process/tooling structure projects in ML can go successfully into production.
The having constant SRE support is a big deal (your point 2). While we had broad ownership, autonomy, and fulfilled many of the 12 factors described in the blog (and shipped ml projects to production), only recently did we have SREs supporting us continuously and it is a night and day improvement in QoL. It is as you said, so much time is fighting your own infra and it just takes a lot of time and mind space.
It really breaks down to acknowledging how many overlaps ML has with traditional software development. Great to hear that adding SREs worked for you - how did the transition go for the rest of the team? Any big learnings from your team about the transition?
Clear the decks of ML engineer time. Your ability to get value from ML engineers directly relates to how much unblocked autonomy they are given. If they are overloaded with bureaucratic chores or maintenance tasks or on-call responsibilities, it is just directly subtracting from their comparative advantage of conducting model training and optimization.
We aren't even close to understanding even this basic concept in some organizations. Sometimes when a data scientist walks into a job in some industries, this is what they are dealing with:
"What's an ML engineer? What do you mean we need to invest in data infrastructure? Email Bill over in that other department and just ask him to send you the Excel book like everyone else..."
To me these patterns feel very close to what organizations have been tackling for the last 20 years in software development. Teams need to be given the autonomy AND responsibility to own their infrastructure, software and processes - especially since this field is moving so fast and still so new (compared to 40+ of software development).
We have a great starting point if we don’t start at 0, but from where the DevOps mindset currently stands.
I couldn’t agree more. The space of ML and MLOps is still at a very early stage, and unfortunately it feels as if organizations have to re-learn all the lessons from software development that were learnt the last 20 years. Tools have a great chance here to bridge the gap a bit more harmonious than what we’ve seen in classical software engineering.
8 comments
[ 2.7 ms ] story [ 43.9 ms ] threadIf you don’t invest in data engineering or you yoke your ML engineers with job duties to also handle all the operations work, you’re not going to see meaningful returns and it won’t be because ML is just hype or is inapplicable to your business. It will because of poor leadership and overloading operations responsibilities.
For ML, the SDLC is just very different.
You need there to exist an auditable system of feature ingestion and serving as a foundation so that model training can standardize, and you don’t overwhelm an ML team with duties to constantly deal with schema changes, data ingestion job delays or outages, collating poorly maintained data across multiple teams.
You also need a standardized way to define training jobs, with easy config interface to specify all the resources needed (GPUs, disk space, cached data sets, certain CPU properties, docker container with the training environment prepared) that connects to an experiment management system to track models and map the entire bundle of parameters (hyperparameters, version control commit hash, resource settings, etc) to each trained model output artifact and evaluation metrics.
If you leave this stuff as free for alls for ML engineers to figure out, they will have to spend so much time fighting with infrastructure that is not their problem that either they’ll burn out and quit to get a job that doesn’t treat them this way, or they’ll resign themselves to being an “ops monkey” and you’ll see an endless churn of infra projects that are supposed to finally unlock the real potential of ML for your business but inevitably never do.
Practices of healthy companies that lead to real value from ML projects:
1. Make data producers responsible for their data delivery, data quality and data timeliness. If some team owns a search page, they also own everything about ingesting the data generated by that page. Data Platform teams might give them tools or provide infrastructure for it, but that application-specific team is ultimately responsible.
2. SRE teams do not helicopter in with parochial best practice recommendations and then leave. SRE teams must co-own the challenging components where the rubber meets the road and really be down in the implementation weeds - this is critically true for unusual or long-tail special case systems, for example like GPU workflows or Spark infra for ML jobs. SRE cannot be either siloed to focus only on the biggest use cases or acting like they just dole out philosophy and best practices. They have to be tied to the delivery incentives of every team they serve.
3. Clear the decks of ML engineer time. Your ability to get value from ML engineers directly relates to how much unblocked autonomy they are given. If they are overloaded with bureaucratic chores or maintenance tasks or on-call responsibilities, it is just directly subtracting from their comparative advantage of conducting model training and optimization. Nobody likes to admit it, but you can get other people to do the maintenance, the on-call shifts or the compliance tasks. Many engineers won’t be sacrificing a comparative advantage by being allocated to that stuff, but if you allocate ML engineers to it, you are flushing money down the toilet by wasting their comparative advantage. Remember: it’s a business - it’s not about what’s “fair” in terms of everyone sharing eg an on-call shift, it matters what creates the best result for your customer.
Before you can apply the tactical steps like the 12 steps of this article, you have to resolve much bigger issues of engineering culture to reflect the 3 points above.
If internal engineering politics won’t block you from these culture changes, then you can set things up to get value from ML. Otherwise, you’ll probably just waste money and have high turnover on the ML teams beca...
We aren't even close to understanding even this basic concept in some organizations. Sometimes when a data scientist walks into a job in some industries, this is what they are dealing with:
"What's an ML engineer? What do you mean we need to invest in data infrastructure? Email Bill over in that other department and just ask him to send you the Excel book like everyone else..."
We have a great starting point if we don’t start at 0, but from where the DevOps mindset currently stands.