I agree, for simple things, FastAPI is really nice to work with. I'm currently rewriting some Flask/Sanic projects with FastAPI to make it easier to maintain.
You are right! I like Django admin panel, without coding you get UI and I'm comfortable with Django + DRF.
In Flask you need to add many things to make it usable.
If you check the code of the tutorial you will see that it differs from the most of the ML-to-REST-API tutorials. It's not only setting the endpoint. You can have many algorithms with many versions and stages (testing, production). There is A/B testing example. Of course you can do all in Flask but I prefer Django because many things are there already.
In our case we've already got all the stuff for serving web pages out of Django... the data model, DRF, auto scaling, access control, CI/CD. If I can use the same infrastructure to serve predictions, without having to duplicate anything or learn new frameworks, that's a huge win. Dealing with CPU overhead is easier than dealing with brain overhead.
Calling the ML model should be done via Django Channels or a similar background queue and not directly within the view (request handler) as if the inputs/inference is too big, it will block and cause issues for load handling.
I agree. The ML Algorithms and models can run into huge sizes. I have one with half a gig of serialised model, when running takes up close to 3.5 Gigs. This approach of directly using it in the views would pose scaling issues.
It really depends. For sure there are models which should be processed in the background. It is a very basic tutorial that shows how simple ML models can be used as REST API. In the tutorial I was using Random Forest trained on UCI Adult Income data set. The final model was small, and computing predictions was fast.
I would love to add more advanced part of the tutorial with background processing for large models.
Inference should be fast, so an async view (which we're excited finally happened!) should be fine. It's similar to waiting on a db. Read consistency means using similar memory. Likewise, you want it to be warm, not reloaded between calls.
In our case, we do on-the-fly GPU calls, with weird spiking behavior, so been looking for a good serverless GPU solution for this kind of stuff.
Depending on the underlying DB, these might be handled the same way at the DB level. For example, Postgres uses the same field type for both CharField and TextField.
A Django app for managing training data and active labeling would be a nice complement to this. Something similar to AWS GroundTruth.
We should start thinking about an ML life cycle were data is ingested, data labeled labeled, models trained, model tested, model deployed and monitored. Rinse, lather, repeat.
For anyone who is looking to make deploying ML models easier, I created https://deepserve.ai
You can run `deepserve deploy` on the command line and it will stand up a REST endpoint for your model. I'm also building client side SDKs to make it easier to call on these models in your application code.
It's still in beta but feel free to reach out to me jeff @ deepserve.ai
1. No it's not open source, but I'm not excluding it from the roadmap :) The selling point is really the easiness of it.
2. Yes! We will be releasing tokens soon so that anyone can interact with the API.
3. Models can be up to 1GB, each model gets their own server , we only do real-time predictions for now. Meaning the models are constantly running waiting for a request
[Gradio](gradio.app) can save a lot of time to serve the same purpose - making it super easy to serve any python based ML. It provides dozens of interface components, including text, image, audio, files, and dataframes to support many different types of predictions.
23 comments
[ 3.1 ms ] story [ 60.2 ms ] threadFor just a REST ML model endpoint, Flask does everything you need, and no more. It's a perfect fit.
If we are talking speed and simplicity then I would father go with Fast Api rather than flask.
In Flask you need to add many things to make it usable.
If you check the code of the tutorial you will see that it differs from the most of the ML-to-REST-API tutorials. It's not only setting the endpoint. You can have many algorithms with many versions and stages (testing, production). There is A/B testing example. Of course you can do all in Flask but I prefer Django because many things are there already.
I would love to add more advanced part of the tutorial with background processing for large models.
In our case, we do on-the-fly GPU calls, with weird spiking behavior, so been looking for a good serverless GPU solution for this kind of stuff.
We should start thinking about an ML life cycle were data is ingested, data labeled labeled, models trained, model tested, model deployed and monitored. Rinse, lather, repeat.
You can run `deepserve deploy` on the command line and it will stand up a REST endpoint for your model. I'm also building client side SDKs to make it easier to call on these models in your application code.
It's still in beta but feel free to reach out to me jeff @ deepserve.ai
It's a simple drag and drop to deploy tensorflow, scikit, spacy, keras or pytorch.
1. Is it open-source?
2. Can I add models with REST API?
3. How do you handle large models? Are they working in the background or you just use larger severs?
2. Yes! We will be releasing tokens soon so that anyone can interact with the API.
3. Models can be up to 1GB, each model gets their own server , we only do real-time predictions for now. Meaning the models are constantly running waiting for a request