Ask HN: What is your ML stack like?
How did your team build out AI/ML pipelines and integrated it with your existing codebase? For example, how did your backend team(using Java?) work in sync with data teams (using R or python?) to have minimal rewriting/glue code as possible to deploy models in production. What were your architectural decisions that worked, or didn't?
I'm currently working to make an ML model written in R work on our backend system written in Java. After the dust settles I'll be looking for ways to streamline this process.
133 comments
[ 4.8 ms ] story [ 213 ms ] threadAlthough in one case we had very tight latency requirements (ie: 10ms) so the ML results were pre-computed and loaded from a cache on the backend servers.
I'm not very sure what you mean when you say the ML results were pre-computed?
>I'm not very sure what you mean when you say the ML results were pre-computed?
We were scoring ads per page, and possible values for both were known ahead of time. So for each ad-page combination we generated the scores and then pushed them into a giant cache.
A proper ML stack is something like:
- Data format in X schema
- Model trained on Y library/platform
- Evaluated and tested using Z
- Serialized in A format
- Stored on cloud B
- Deployed using C
- Versioned using D
- Real-time monitoring using E
I contract for some clients in fintech and some defense-related stuff.
EDIT: Took a little digging but I found it [1] (and yes it's primarily x86_64 linux+macOS)
[1] https://github.com/crystal-lang/crystal/wiki/Platform-Suppor...
What are the popular use cases - either ones driving current development or ones where it's best suited?
Occasionally I will use rust or C/C++ for some of these tasks, but I try to keep things in crystal whenever I can.
Shipping pickled models to other teams.
Deploying Sagemaker endpoints (too costly).
Requiring editing of config files to deploy endpoints.
What did work:
Shipping http endpoints.
Deriving api documentation from model docstrings.
Deploying lambdas (less costly than Sagemaker endpoints).
Writing a ~150 line python script to pickle the model, save a requirements.txt, some api metadata, and test input/output data.
Continuous deployment (after model is saved no manual intervention if model response matches output data).
https://docs.python.org/3/library/pickle.html
https://docs.python.org/3/library/pickle.html https://scikit-learn.org/stable/modules/model_persistence.ht...
We use dill[0], but there are other similar libraries.
[0] https://pypi.org/project/dill/
Basically everything you work with in programming is a value (the number one-hundred-seventy-five, for example: "175") or the address—location in computer memory, say—of a value. You might record the address of that value above as the count of characters from the beginning of this post, for example, were this post the layout of data in some RAM, just as numbering houses on a street. Add the concept of data "width"—how long the number is, in terms of how many characters represent it (3, in this case) and you've got basically all there is in terms of primitive stuff that computer programs operate on.
Observe that the value stored at a location in memory, say, might itself be an address—location—of some other thing stored in memory.
The width+value concept can get you pretty far, in that you can store a bunch of stuff and find it again, given the address of the beginning and some convention that the first so-many bits of the value describe the width of the rest of the value, or some other means of knowing the size (width) of the value, as long as all its parts are stored right next to each other in memory, and in the correct order. That's called an array, in fact, which is a data structure! One problem with arrays is that if you want to make them longer, you might not have more memory available at the end of them—something else may be using that location already, and if you just overwrite it that'll likely break something. So you'll have to copy your entire array to a larger piece of empty memory to add on to the end of it.
EXAMPLE!
Say we have some RAM large enough to store nine things, and we know that we have something stored at position 4—programmers like to order items starting with zero, but I'll refrain because it's not really important here and makes it more confusing. The RAM contains the stuff we're looking for, plus some other stuff that we don't care about right now:
[429317501]
We look at position 4, and know (by convention, or whatever) that the "3" we find tells us how many more locations to read past that to get our entire value, and extract "175" as our value, by proceeding to do just that. That's an array! One of the most basic data structures. Of course all this is binary under the hood, and those binary numbers can also represent letters or color values of a pixel in an image or whatever, but I'm using simple base-10 numbers to keep things easier to follow.
You can use these basic pieces to build up more complex data structures than that, of course. You can have a series of places in memory, not necessarily next to each other, each containing a value and then the address of another value+address pair. Given the address of the first of these, one could write a program to read each in order, following the addresses to hop from one to the next until it finds one without an address provided. Ta-da, it's (one kind of) a linked list! That's another kind of data structure. Now if you want to add to the end of your value, you just pick an empty spot in memory, add its address to the last piece of the existing list (first "walking" the list to find it by starting at the beginning and reading each piece in turn, if you don't already know where the last part is located), then fill it in with the value you want. This takes up more space than an array, though, and may take a little longer to "read" (get the value[s] from). It's very common for many types of data structure to be able to represent the same thing, just with different trade-offs in terms of space used, or time to locate a given part of the structure, or how much space or time it takes to modify it (recall, having to copy an entire array in order to add on to it) and so on.
EXAMPLE!
[950818972]
If we know (somehow) that our linked list starts at position (address) 5, and know that we can expect a value then an address at that location,...
Any other clear descriptions of CS concepts - especially for Data Science - to share ? Links to them ?
1. The universe is composed of things.
2. We can use the computer to store information about those things (this is the data).
3. In order to gain useful insights about those things, we want to do operations on the data. I.e. to compute. computation is done by algorithms.
4. Data structures are the bridge between the data about things and the algorithm. They hold the data such that the algorithm will have an easier time computing.
It packages your model for you into a standardized format, that you can use it in multiply serving scenarios online serving with api endpoint, offline serving with spark udf, CLI access or import it as python module. It also helps you deploy to different platform such as lambda, sagemaker and others.
Our value is from model in notebook to production service in 5 mins. Love to hear your feedback on this. You can try out our quick start on Google colab (https://colab.research.google.com/github/bentoml/BentoML/blo...)
BentoML looks more cohesive than our homegrown solution because it targets a more general case. One of the things I would miss switching to BentoML would be automatic requirements generation. We use pipreqs[2] to generate a requirements.txt given a model instance. Any thoughts on the difficulty as a user in extending BentoML as to integrate pipreqs?
Again another difficulty question: we have a few statsmodels[3] predictors and it isn't clear how much work would be involved extending BentoML to accept those too.
Thanks for pointing out BentoML. I'll keep an eye on it as a migration target as this space develops.
[1] https://mlflow.org/docs/latest/index.html
[2] https://github.com/bndr/pipreqs
[3] https://www.statsmodels.org/stable/index.html
It should be very straightforward adding support for saving/loading Statsmodels in BentoML. In fact you should also be able to just use the existing "PickleArtifact" in BentoML for statsmodel predictors too. We will add an example notebook for working with Statsmodels library soon!
[1] https://kubeflow.org
[1] https://github.com/awslabs/aws-sam-cli
If we were to use Cortex, we would likely wrap the creation of cortex.yml in a function and call it when we're saving our models. We do something similar right now and store the meta in json files for later deployment. I love tracking config in git too.
pipenv
https://packaging.python.org/tutorials/managing-dependencies...
https://dev.to/yukinagae/your-first-guide-to-getting-started...
1. https://news.ycombinator.com/item?id=18612590
Point about "Official tool" is valid
Others seems strange to me
I'd add to my comment something like "Try it. If you like it - use it"
Thank you for link
FastAPI for quickly creating new API endpoints. It has automatic _interactive_ docs and super simple data validation via Python typehints, so that we don't waste compute time with malformed data. https://fastapi.tiangolo.com/
We deploy on prem most of the time, but have started using GCP on occasion.
0: https://github.com/shark8me/clj-ml
1: https://github.com/uncomplicate/bayadera
2: https://www.clojuriststogether.org/news/q4-2019-funding-anno...
[1]: https://github.com/eclipse/deeplearning4j
Here are the artifacts we produce:
1. For new models we often build a demo endpoints/glue code written in python/flask that can be compared against the prod output in dev/psup.
2. Deep learning models (much of what I do personally): saved in TF saved model format. If it is an update to an existing model often it is just a drop-in replacement. If it is a brand new model i will often include a flask demo (the python code does proper data transformation before calling on tf). On production side, after testing/regression these model are deployed via tensorflow-serving containers and used as gRPC endpoint. For production, whatever data pre-processing needs to be done is written by the backend team, who compare preprocessing output with our demo.
3. Logistic regression/tree models: again, for new models we provide the demo but what goes into production are either csv (logistic regression) or json (tree) of the weights/decision boundaries which are used as resources by the backend team's Java code.
The overall flow is:
ETL (via apache airflow/custom code) => model training/feature engineering => (saved model file + flask demo endpoint/documentation on feature transformations) => dev incorporate model/test into java backend => comparison of demo vs java backend => regression of java backend (if they had previous versions of model) => psup (small amount of prod data duplicated and ran in parallel with prod) => prod (model deployed + monitored)
There is a caveat that we also do some batch processing/not really live analysis that is just done in python and then results are pushed wherever they need to be pushed. In this case we don't involve the backend/java team.
It also provides OpenAPI spec for your API endpoint, which allows you to generate API client in Java, for your backend/app teams.
Disclosure: I'm the top contributor to the project.
Another way to compare is as follows:
https://www.openhub.net/p/openapi-generator vs https://www.openhub.net/p/swagger-codegen
We are currently looking at MLflow[3] for the tracking server, it has some major pain points though. We use Tune[4] for hyperparameter search, and MLflow provides no way to delete artifacts from the parallel runs which will lead to massive amounts of wasted storage or dangerous external cleanup scripts. They have also been resisting requests for the feature in numerous issues. Not a good open source solution in the space.
Note that this is for an embedded deployment environment.
[1] https://github.com/onnx/onnx
[2] https://github.com/Microsoft/onnxruntime
[3] https://mlflow.org/
[4] https://ray.readthedocs.io/en/latest/tune.html
While I never made my research code public, https://openreview.net/forum?id=SyxytxBFDr documents a similar setup to what I use.
We have a team of 6 DS working on parallel projects. We tried the Java service approach. It's great for a one-time model, but very painful to iterate on.
We develop on top of Sagemaker, and since we're a funded company, can somewhat get away with the 40% price increase of an "ML Instances".
We have a mix of R/Python models. For each, we keep a separate repo with a Dockerfile, build file, and src code.
Training:
If the jobs are small, we train them locally, package assets into the container, and deploy. If it's a bigger job, we leverage Sagemaker training jobs and S3 for model storage.
Serving:
We have boilerplate web service layer with an entrypoint that DS fills in with their own code. Yes, this allows almost arbitrary code to be written, but we do force code reviews and enforce standards. Convention over configuration.
We do the feature engineering using Python/R, which when parallelized, has good enough performance (sub 200ms latency on Sagemaker prod). If we need latencies in the 1-10ms range, we'd consider refactoring the feature engineering into a separate layer written in a more performant language. It's always FE that takes the most time.
One learning from tuning Python services is: for max performance, try to push the feature engineering work onto consumers. Have them fully specify the shape of the data in a format that your models expect so you do as little feature engineering in the serving step.
Training:
Currently we just use a bunch of beefy desktop workstations for training (using Pytorch).
Deployment:
This is the vast majority of our cost, each time a paraphrase comes in we add it to a queue through google cloud Pubsub. We have a cluster of GPU (T4) servers pulling from the queue, generating paraphrases and then sending the responses back through Redis pub/sub. I think ideally we would have a system that makes it easier to batch sentences of similar length together, but this seems to be the most cost effective way for models that are too computationally expensive for the CPU that is relatively simple to put together.
Models and feature engineering done in python, trained locally, weights uploaded to S3. Dockerfile with a tiny little web server gets deployed through or CI/CD pipeline for serving.
Soon: Argo workflows + Polyaxon for data collection, feature engineering, training etc. Push best model tobS3, same CICD process with docker container deploys little web server onto our Kubernetes environment.
Deep learning stuff will probably use a similar setup, but with PyTorch instead of Sklearn. Would like to look at serving with ONNX exporting.
When the Julia packages evolve a little more, will be looking forward to using that in production.
Happy to answer any question or provide more information.
[0]: https://github.com/polyaxon/polyaxon
[1]: https://github.com/polyaxon/polyaxon/blob/master/cli/tests/f...
I thought MlFlow was a spark thing, and were trying to migrate off of spark/DataBricks due to the resources inefficiencies of Spark (at our scale) and maintenance nightmare that python notebooks are causing us.
Argo is just a container workflow tool, not ML specific. We’re planning on using Argo for the data engineering parts, and polyaxon for the ML training parts because of the convenient monitoring and hyper parameter search tools.
[1] https://github.com/kubeflow/
Argo workflow is a pipeline engine that is cloud and kubernetes native. It tries to solve graph and multi-steps workflows using containers on Kubernetes, It can be leveraged for ML pipelines as well as other use-cases.
Kubeflow is a large project that has several components: training operators, serving (based on Istio and Knative), metadata (used by tensorflow TFX), pipelines, ... and integrates with other projects. Kubeflow pipelines is using Argo workflow as a workflow engine, although I think there are efforts to support other projects such as Tekton which is also a google project, and possibly TFX as a DSL for authoring pipelines in python.
The main focus for MLFlow, I think, is tracking ML models and providing an intuitive interface to model deployment and governance. The main strength of MLFlow is that it's easy to install and use.
Polyaxon has been used mainly for fast developement and experimentation, it has a tracking interface and several integrations for dashboarding, notebooks, and distributed learning. Polyaxon also has native support for some Kubeflow components, e.g. TFJob, Pytorch job, MPIJob for distributed learning.
The upcoming Polyaxon release will be providing a larger set of intergrations for dashboards, in addition to tensorboards, notebooks and jupyter labs, users will be able to start and share zeppelin notebooks, voila, plotly dash, shiny, and any custom stateless service that can consume the outputs of another operation.
The new workflow interface focuses mainly on an easy declrative way to handle DataOps and MLOps, the main idea is to provide a very simple interface for the user to go from a data transformation to training models. Since the component abstraction is based on containers, it can be used to do other operations, e.g. packaging models and preparing them to be served on other open source projects, cloud providers, or lambda functions. Also support for some frameworks such as dask, spark and flink operators could be used as a step in a workflow, ...
For hyperparams tuning, Currently, the platform has grid search, random search, hyperband, and bayesian optimization, one of the major changes in the next release is a new interface for people to create their own algorithms and a mapping interface to traverse a space search provided by the user or based on the output of another operation.
From Polyaxon's side, we will be providing a set of reusable components, some of these components will be targeting deployment and packaging, for instance aws lambda, azure ml, sagemaker, open source projects for deployment (some of them are mentioned in this HN thread) Users can also contribute components as well or create them to be used inside there organization.
For versioning, all component have versions, and all runs have full overview of their dependencies and provenance (inputs/outputs). This gives all information about when and how a model was created.
The platform knows how to create and manage services, e.g. tensorboards, notebooks, dash, simulators for RL agents ..., so one can also deploy the model as an internal Polyaxon service to be used as an internal tool or for testing purposes, the only thing to keep in mind is that the API endpoint will be subject to the same access rights as other components create by a given user.
When we do need A/B testing, we’ll probably use something like Seldon. As for predictions/second, not very much at the moment: 1 per 30 seconds maybe? It’s not deployed into a Kubernetes cluster because of scaling requirements, it’s because that’s where all our other services greet deployed till, and it’s more beneficial (ops and cost wise) to also deploy into there than it is to bother with having a separate workflow for deploying to lambda’s or SageMaker.
Also, it looks like you have a very low volume of predictions?
One dead simple way to do this (R model —> Java production) that I’ve done in the past is to use PMML (via pmml package), which converts models to an XML representation. ONNX is a similar/newer framework along these lines. You can also look at dbplyr for performing (dplyr-like) data preprocessing in-database.
PMML is language agnostic model specification (XML like). Python and R machine learning ecosystem can easily generate these (caveat, only tried for gbdt and linear models and not sure this works well for neural nets).
Openscoring is Java library that creates rest API for scoring models. It's lightweight, battle-tested, nice API, good model versioning and in my experience 10x faster than Python flask. You don't need to write any Java code, just download and run the .jar and post valid PMML to the right endpoint.
Another feasible approach is Sagemaker deploy - code from Jupyter notebook can deploy API in one line. I think this can be less economical and have higher latency if you will have high usage but a datascientist can do model updates from within a notebook.
Please NEVER hardcode regression model coefficients within Java. This is a nightmare to maintain, prevents increasing model complexity and is no simpler than PMML + openscoring. I think you can wrap the Java PMML library in another Java web framework like spring if you need something more bespoke.
https://www.rdocumentation.org/packages/pmml/versions/2.1.0/...
https://github.com/openscoring/openscoring
https://aws.amazon.com/blogs/machine-learning/using-r-with-a...
> Please NEVER hardcode regression model coefficients within Java.
Amen to that.
https://scikit-learn.org/stable/modules/generated/sklearn.pi...
The bit I'm not sure about how to do well is model monitoring.
Jobs are scheduled (Azkaban) for reruns/re-training and pushed from data env to the feature/model-store in live env (Cassandra). Online models are exported to SaveModel format and can be loaded on any TF platform, eg java backends.
Online inference using TF Serving. Clients query models via grpc.
A lot of our models are NN embedding lookups, we use Annoy for indexing those.
I appreciate that this approach won't work for everyone.
Most of our tools are built in Rust. Several of those are for creating(cleaning) data streams out of datasets. They are converted into tensors or ROS messages.
Models are written in Python (mix of pytorch/NLP/tensorF). The Models are serving about 35 predictions/second on avg. The API server written in the Python. API server container feeds or write the requests in the distributed queue cluster. The models picks up the samples from the queue in batching. It allows to experiment the models (different flavor) based on the routing being set during the deployment time and which in turns being set in the cache. We use AWS managed cache, queuing and container orchestration platform. Next: 1)Current pipeline for the training and production is two separate pipeline which we want to combined, possibly use MLFlow, Airflow or KubeFlow. Deployment to the production is done through Jenkins. 2)Active retraining and auto deployment to production. 3)Tie the version of model in production to model being trained. There is no way for us to tie back the version.
- CTO (chris at vetd.com)
Data scientists have a similar docker image running in kubernetes which includes all of these images as conda environments for experimenting in prod-like environments. Spark is used to fetch data for the most part.
Models report a finished state over Kafka after getting persisted to buckets in Google cloud, then gets mirrored over to a ceph cluster connected to our serving kubernetes cluster.
We have an in house Golang server binding to c++ for serving pytorch neural nets persisted with the torch.jit API (I can really recommend this for hassle-free model serving). We also have some Java apps for serving normal ALS or Annoy based models.
Our traffic is not as wild as many here, but we're serving around 10M user requests a day.
We also do a merging of results from several models' results, and join them together with a separate "meta-model" that estimates which model the user has had a preference for recently, to weight those up.
There's probably a lot of details left out here, especially about the serving part, since we have various services in front of the models enriching data and presenting it to the user, but it's the gist of it.
Models:
- Models are structured as python packages, each model inherits a base class
- base class has define how to train, and how to predict (as well as a few other more specific things)
- ML engineer can override model serialization methods, default is just pickle
Infra:
- Code is checked in to github, Docker container built each merge into master
- Use Sagemaker BYO container to train models, each job gets a job_id that represents the state produced by that job (code + data used)
Inference / deployment:
- Deploy model as http endpoints (SageMaker or internal) using job_id
- Have a service that centralizes all score requests, finds correct current endpoint for a model, emits score events to kinesis, track health of model endpoints
- A/B test either in scoring service or in product depending on requirements
- deploy prediction jobs using a job_id and a data source (usually sql) that can be configured to output data to S3 or our data warehouse
So far this has been pretty solid for us. The tradeoff has been theres a step between notebook and production for ML engineers which can slow them down, but it forces code review and increases the number of tests checked in.
This was a game-changer for us. What does your testing story look like?