HN Explorer

24 comments

[ 6.4 ms ] story [ 71.8 ms ] thread
jbverschoor
I have absolutely no idea what this is, supposed to do, and what the demo shows.
nitelord
From the Github: End-to-end machine learning project showing key aspects of developing and deploying real life ml driven application ... a web based ml driven bike trip advisor with trip time prediction
t0astbread
From what I can tell it's an example application to show how a machine learning project could be turned into a real world web app more or less from start to finish.
julienkervizic
Yes it covers a bit more than what people tend to see in Machine Learning which is the model training part. From the code base it looks to cover:

1) feature preparation and model training as part of notebooks

2) Creation of a Flask API to interact with the trained models. This includes feature enrichment based on APIs input, so as to match the expected inputs by the model

3) Creation of an UI to interact with the API/Model

4) Setup of NGinx and docker to surface that application

Each part is covered in a minimal manner compared to most enterprise data products, and only cover 1 of the approach for end to end ML, but I think it does the job well to demonstrate the scope of work that is needed to put ML data products into production, and could be used as good introduction.

discordance
To get ML models into production is a lot of work involving tests, retraining, model iteration/management and model serving (often at the edge where latency is low).

ML-Ops is as deep as data engineering but not as often talked about. The linked article discusses one example.

dotdi
I came here because I thought this was about the ML programming language, aka Lisp with Types™, but it's about machine learning.
taneq
The TLA namespace is badly overloaded enough, but it's got nothing on the TLA namespace.
tasogare
I made the same mistake most of the time but here with notebook in the title is was obviously Machine Learning.
jetti
I made the same mistake for this post. I'm not very familiar with Machine Learning so notebook doesn't give it away for me. But then throw in "fullstack" and it makes it seem, to me, that it is a front end/backend web framework written in some variant of ML
[deleted]
(comment deleted)
[deleted]
(comment deleted)
leetrout
This is very, very similar to what Yhat (YC W15) did.

You dropped some code in your Python or R project and it would slurp everything in to one package and, for Python, run it in Flask in a docker container using their home-grown container orchestrator.

https://www.welcome.ai/tech/data-science/yhat/

s3nh_
Looks really nice! Love to see it next to some Rust-like or Golang=like backend. What do you think?
olooney
This feels like it doesn't live up to its own hype because it doesn't really exhibit a production-quality deployment of a Python application but something closer to a dev server. It uses a single-threaded flask server behind a nginx reverse proxy without a WSGI layer to handle multiple requests, etc. Here is the nginx code:

    server {
        listen 80;
        server_name: localhost;
        location / {
            proxy_pass http://fai:4242/;
            proxy_set_header Host "localhost";
        }
    }
According to the docker-compose.yml file, fai:4242 is running a container called "flask":

      flask:
        build:
          context: ./
          dockerfile: Dockerfile
        container_name: flask
        networks:
          net:
            aliases:
              - faip
And according to the Dockerfile for the "flask" app.py runs flask built-in (single threaded, recommended for development only) server:

    if __name__ == '__main__':
        APP.run(host='0.0.0.0', port=4242)
There's not much to say about this except that the flask project themselves don't recommend running this server in production[1]. I would recommend a WSGI server such as gunicorn[2] running under supervisord[3]. I would also like to see some logging and error handling in production quality code, and it would make sense to have nginx serve the static files (index.html, index.js, etc.) directly without using up a python thread (which should be reserved for the more dynamic endpoints.) It would be nice to demonstrate using nginx for rat limiting[4] or as a load balancer[5] spreading the work across multiple flask servers.

Mind you, I think this is a neat little demo, but as something to put into production - or to teach other people how to put REST APIs into production - it falls short in its current state.

[1]: https://flask.palletsprojects.com/en/1.1.x/tutorial/deploy/

[2]: http://docs.gunicorn.org/en/stable/deploy.html#deploying-gun...

[3]: http://supervisord.org/

[4]: https://www.nginx.com/blog/rate-limiting-nginx/

[5]: http://nginx.org/en/docs/http/load_balancing.html

astonex
I don't think supervisord is necessary when you're in a container environment. Just write a healthcheck to bring up a new container when something goes wrong
m23khan
this is freaking awesome! was looking for something like this for a long time. Much appreciated!!!
[dead] CorrodedCorr
Nice demo! I definitely agree that the experience of building a real-world products out of ML-related ideas is what often differentiates juniors from seniors. Building sklearn models out of clean data can only prepare you so much for the real life.

Luckily people are trying to address this gap in education options! For example, https://sharpestminds.com is a platform where you gain skills by building such a full-stack ML project under a mentorship of an experienced data scientist yourself.

jkaufmann_
While the example here is definitely trivial, it would be nice to see some more documentation. Particularly around project structure, internals, and thought process.

Seeing e2e projects is definitely a huge plus, but without relevant commentary I don't see how it is consumable enough to help a large enough mass.