As one of the main drivers of Python at Netflix, this one is near and dear to my heart. Let me know if you have any questions, or come visit our booth!
There was many interesting things listed at the blog post. I have been planning a system similar in idea to Sting, your OLAP server (?).
I'm planning a light RPC style system on top of Pandas dataframe objects, transferring HTTP requests to Pandas method calls. Memory usage of such a system with our data is too expensive if all data is in use. So, I'm investigating possible caching and database solutions.
Could you perhaps elaborate on the implementation of Sting's in memory database?
I just posted this above: "I don't have any info about Sting, but we're working on some blog posts that go deeper into each of these topics. I'll make sure that the interest in Sting is accounted for in the prioritization."
> It's tough as everyone is drinking the ruby kool-aid.
Ruby and Python are similar in terms of execution speed and developer productivity. If your work requires something that Python has and Ruby doesn't(say numpy), then it might make sense; otherwise what would you accomplish by switching from Ruby to Python?
Posts like these are great on a tech level, but they're also good recruitment - Seeing Python, along with hackers that I respect (like you) at Netflix make me at least the littlest bit tempted to go there ;)
But in all seriousness, we like to share what we do because sharing is cool and it attracts other people who like to share cool technology, which are the kind of people we like to work with.
We were actually talking about your company in our sprint planning meeting a few minutes ago, since we're doing in-house Python analytics. You might be getting a call.
Awesomeness. We are making great headway on Blaze and the new persistence format to supplant HDF5/PyTables (Although, Blaze will always work great with carray, NumPy, and Pytables), and the Numba compiler project is working out great.
Wakari (http://wakari.io) is nearing 1.0 and we're about to turn on paid monthly plans for beefier nodes and GPU nodes, and we have also been in talks with several companies about in-house private deployments.
Give us a shout or shoot me an email at pwang (at) continuum.io! :-)
You know how Google claims that by saving fractions of a second for every search that billions of people do, they've saved many man-lives of productivity? As a leading engineer at both Reddit and Netflix, you've undone all of that. Well done.
I don't have any info about Sting, but we're working on some blog posts that go deeper into each of these topics. I'll make sure that the interest in Sting is accounted for in the prioritization.
Kurt Brown had some slides about data & Python at the NYC Python meetup last October. He did a Sting demo there and some of the slides talk about it (I think... it might be a different thing, though. October was a long time ago..)
I have to say that after doing a python project I totally don't get why people love it so much. The syntax is slightly cleaner, but the lack of threading, first-class functions and dependency management tools make it waste at least as much time as it saves. I honestly don't believe in the concept of a programming language being able to make code more or less readable. A Java project with a POM file can be up and running on any OS in about 5 minutes.
I often get the idea that people confuse anonymous and first class functions (apparently because they discover the latter expressed in terms of the former).
a) python-pip combined with virtualenv gives very good requirement support.
b) python does have first class functions?
c) threading still happens, you just can't take advantage of multiple cores. This isn't as big of a problem as you'd think though, you can still write very fast code.
Sorry, I should have said "lack of true threading in CPython" by which I mean the GIL limitation. And lambda expressions are lame compared to what you can do in JavaScript/Perl/Scala. And the build tools suck compared to Maven which is saying a lot. Maven is cross-platform and install-free. It takes a long time to makes head and tails of pip, easy_install and virtualenv and it still does less than Maven.
>And lambda expressions are lame compared to what you can do in JavaScript/Perl/Scala.
Python lets you pass functions other ways than using lambdas, you know. If you've defined a function with the standard "def foo" style, you can pass "foo" as an argument.
Even the standard "1-line lambdas LOL" limitation can be circumvented by defining the function elsewhere and making the lambda a call to it.
What do you use "true" threading for? Are you multiplying matrices? If so use Numpy. Are you downloading data and accessing database? If so, you threading, threading in Python works great for that. If you hit memory size issues, switch to gevent or eventlet for green threads.
I have been building large projects in Python for the last 10 yeas and never really thought "Hmm I wish I had more powerful lambda functions". You can always just have regular function and pass that around. Why do you need lambdas so much?
Lambda expressions are not necessarily first-class functions (I mean, they are in Python, but so is any other old function).
First class functions simply mean that a language treats functions as it would nearly any other data type, namely that they may be passed as parameters, bound to variables, and returned from other functions.
Python's poor handling of lambda expressions do not invalidate that functions are first-class citizens.
It has its share of warts, but doesn't lack any of those things you mention.
Java shines at the huge end, but for most everything else python is a more productive choice as it is so much more concise. Perhaps you are not yet using it optimally?
As a Java/Python shop I think Netflix should evaluate Go after 1.1 RC comes out in April. Go is very popular with people who have heavily used both Java and Python (as well as C).
If I had to guess, I would say because GP didn't actually give any real reasons why adding complexity (a new language) would improve or solve a problem. Advocating languages is fine, we all have our pet languages, but it is important to remember they each are just one tool in the toolbox and that what is more important is how the language solves the problem well, instead of just blindly advocating one language over another.
Language comparison are so ridiculously multi-dimensional and my knowledge of Netflix's internal applications are shallow enough I thought specifics would be a waste of time. I see Go in general as a reaction to the overcomplexity of Java/C++ and the performance issues of Python/Ruby. Since Go seems to be popular with people who use Java/Python, and Netflix uses Java/Python, I was curious if Go was on the table. I would hardly call my original post blind advocacy...
Some folks would consider this as an attempt to hijack the discussion. I am not saying that is what you are trying to do but what some folks would think. It gets tedious sometimes when the post is specifically about "Python at whatever" and people post "Hey you should use this language instead."
Maybe that is why you were down voted. (FWIW, I did not down vote you.)
I posted this: http://news.ycombinator.com/item?id=5356913 today as a reply to someone who commented about Go on a post about programming languages today. It seems that (at least so far) I was correct.
Besides Google using Go for some things like part of Youtube and dl.google.com, CloudFlare, Heroku, Canonical, Torbit, SoundCloud, BBC, and others are using Go for parts of their back-end infrastructures.
Because it's not helpful to anyone. It just says "they should use my favourite toy instead of their favourite toy". No details about why or how it would help anyone anywhere ever. Just "Someone else should do a lot of work. Because I said so."
I believe we do have one or two people here looking at Go.
Given our culture, there really is nothing stopping anyone from using it, other than the fact that they would have to figure out how to interact with the platform.
I know of at least one person that wrote a fairly significant chunk or erlang at netflix, but this was years ago just as we were moving into the cloud and now, as jedberg wrote, erlang would have a hard time working with the rest of the netflix stack.
I've used go at netflix before to hack up a quick load test, but recently the vast majority of my non java/c++ code at netflix has been python for writing glue to interact with our ops infrastructure and julia, which recently supplanted python/pandas/scikit, for my my research oriented stuff.
But overall go is a really neat "systems" level language.
Re: Web applications in python - I am curious about the overall feeling for them - Do you write web apps in python because rapid development makes up for any performance problems and if you run into problems you would rewrite it in something faster, or do you feel like a python web app can be just as good as alternatives when it comes to speed?
Put another way - when you think of python for web applications / services - do you think rapid development and speed, or just rapid development and you would go somewhere else for speed?
I don't think anybody equates Python with execution speed. Writing web applications in Python facilitates rapid development, and since most of the web applications are io bound, slow execution speed doesn't make noticeable difference.
> when you think of execution speed in web applications
The whole web application is not CPU bound. The parts which are CPU intensive are rewritten in performant language. For eg, if you are twitter, you still render templates in Rails, and write CPU bound services in Java/Scala. Replacing the whole app with Scala/Java will be pointless.
That doesn't mean that every time you need a service, you can't use Ruby/Python. If all your API server does is check cache for data, load data if there is a cache miss, encode json and return it, Python/Ruby will do just fine.
Understood - was just curious when it comes to CPU intensive portions what you would characterize as performant vs python. Scala/Lift is on my list to check out - So my question now is - if you got to the point of needing to write CPU bound services in Scala, why wouldn't you go ahead and write everything in Scala? Does python offer enough advantages to outweigh having to support multiple languages/environments/ecosystems?
I like Python for the speed of development. Also, at least for these internal apps, speed is generally of secondary concern, since these aren't typically high load applications.
At reddit we solved the problem by writing the most often called parts in C and using c extensions. There is definitely more work to do there, but overall I think Python gets a bad rap as far as speed is concerned.
It's certainly not the quickest, but it isn't a dog either.
> At reddit we solved the problem by writing the most often called parts in C and using c extensions.
I'm wondering if anyone has solved these sorts of problems with Cython in the real world. It looks like a great solution for a large set of problems in terms of Python performance, since it becomes a CPython extension, but I never hear anyone talk about it. Can anyone chime in?
I get the impression that Cython is used quite a bit in the scientific computing side of Python, but not so much for web development. pandas & pyzmq are two projects I know that make serious use of Cython.
Cython is used all over the place in some parts of the real world, but you may not hear about those parts because they are not that connected to web app development.
The entire scientific Python ecosystem has basically moved on to Cython. Its use is quite prevalent, and it's basically displacing SWIG as the preferred wrapping tool of choice.
I'm thinking pure speed is not a big differentiator these days... as long as it isn't horrible.
Why? The apps are mostly IO bound, and any large site is going to be caching, deferring slow operations, and sharding/breaking into tiers, cdns, etc, etc. The amount of time a web framework actually runs is relatively small.
Others have already mentioned what to do when performance is a problem.
Python really does hit a sweet-spot for many types of operations. Devops capacity in system scripting, crawling and scraping with excellent web frameworks (Django through to Bottle). On top of all that there are great machine learning libraries as well as good APIs to almost anything not written in Python. My site gets huge amounts of NLP, machine learning and information extraction done in just a few lines because of libraries available in python that aren't in the languages that it's usually compared to.
I've also pretty much replaced bash with the ipython shell at this point, as you can use all of the bash commands anyway and even seamlessly mix them with python, file_list = !ls, it's a excellent environment for data analysis and manipulation.
I have 'ipython --profile pysh' in my terminal's custom command, on servers I'm a bit more cautious and have a 'pu' (power up, or python up) command which is basically the first thing I type. You have to watch out for the shell history not working quite the same and that newly installed programs won't be seen until you do a '%rehashx'. Other than that it's pretty much all gravy. Now I just need to get around to transitioning to that python tiling window manager.
While I've got you, and seeing you are not longer at Reddit, what do you think of jkl.io (launched today)?
I think someone asked the question at a Scipy conference a few years back, "Who here uses IPython as their primary shell", and IIRC about 30% of the people raised their hands.
77 comments
[ 7.2 ms ] story [ 139 ms ] threadThere was many interesting things listed at the blog post. I have been planning a system similar in idea to Sting, your OLAP server (?).
I'm planning a light RPC style system on top of Pandas dataframe objects, transferring HTTP requests to Pandas method calls. Memory usage of such a system with our data is too expensive if all data is in use. So, I'm investigating possible caching and database solutions.
Could you perhaps elaborate on the implementation of Sting's in memory database?
[posted from yet another HN account]
So now that's +2. :)
The idea of marrying "slow" Python and OLAP is disturbing my sleep!
http://databrewery.org/cubes.html
P.S. I really enjoyed your talk at QCon SF last year!
Ruby and Python are similar in terms of execution speed and developer productivity. If your work requires something that Python has and Ruby doesn't(say numpy), then it might make sense; otherwise what would you accomplish by switching from Ruby to Python?
Simpler syntax?
Posts like these are great on a tech level, but they're also good recruitment - Seeing Python, along with hackers that I respect (like you) at Netflix make me at least the littlest bit tempted to go there ;)
But in all seriousness, we like to share what we do because sharing is cool and it attracts other people who like to share cool technology, which are the kind of people we like to work with.
Wakari (http://wakari.io) is nearing 1.0 and we're about to turn on paid monthly plans for beefier nodes and GPU nodes, and we have also been in talks with several companies about in-house private deployments.
Give us a shout or shoot me an email at pwang (at) continuum.io! :-)
http://en.chessbase.com/home/TabId/211/PostId/4008728
http://www.meetup.com/nycpython/files/
b) python does have first class functions?
c) threading still happens, you just can't take advantage of multiple cores. This isn't as big of a problem as you'd think though, you can still write very fast code.
Also, packaging is a whole different can of worms (but hopefully will be sorted by distutils2!).
Python lets you pass functions other ways than using lambdas, you know. If you've defined a function with the standard "def foo" style, you can pass "foo" as an argument.
Even the standard "1-line lambdas LOL" limitation can be circumvented by defining the function elsewhere and making the lambda a call to it.
I have been building large projects in Python for the last 10 yeas and never really thought "Hmm I wish I had more powerful lambda functions". You can always just have regular function and pass that around. Why do you need lambdas so much?
First class functions simply mean that a language treats functions as it would nearly any other data type, namely that they may be passed as parameters, bound to variables, and returned from other functions.
Python's poor handling of lambda expressions do not invalidate that functions are first-class citizens.
Java shines at the huge end, but for most everything else python is a more productive choice as it is so much more concise. Perhaps you are not yet using it optimally?
Hit me up on email (it's in my profile).
Maybe that is why you were down voted. (FWIW, I did not down vote you.)
Given our culture, there really is nothing stopping anyone from using it, other than the fact that they would have to figure out how to interact with the platform.
But overall go is a really neat "systems" level language.
Put another way - when you think of python for web applications / services - do you think rapid development and speed, or just rapid development and you would go somewhere else for speed?
The whole web application is not CPU bound. The parts which are CPU intensive are rewritten in performant language. For eg, if you are twitter, you still render templates in Rails, and write CPU bound services in Java/Scala. Replacing the whole app with Scala/Java will be pointless.
That doesn't mean that every time you need a service, you can't use Ruby/Python. If all your API server does is check cache for data, load data if there is a cache miss, encode json and return it, Python/Ruby will do just fine.
At reddit we solved the problem by writing the most often called parts in C and using c extensions. There is definitely more work to do there, but overall I think Python gets a bad rap as far as speed is concerned.
It's certainly not the quickest, but it isn't a dog either.
I'm wondering if anyone has solved these sorts of problems with Cython in the real world. It looks like a great solution for a large set of problems in terms of Python performance, since it becomes a CPython extension, but I never hear anyone talk about it. Can anyone chime in?
[1] http://sagemath.org
The entire scientific Python ecosystem has basically moved on to Cython. Its use is quite prevalent, and it's basically displacing SWIG as the preferred wrapping tool of choice.
Why? The apps are mostly IO bound, and any large site is going to be caching, deferring slow operations, and sharding/breaking into tiers, cdns, etc, etc. The amount of time a web framework actually runs is relatively small.
Others have already mentioned what to do when performance is a problem.
I've also pretty much replaced bash with the ipython shell at this point, as you can use all of the bash commands anyway and even seamlessly mix them with python, file_list = !ls, it's a excellent environment for data analysis and manipulation.
While I've got you, and seeing you are not longer at Reddit, what do you think of jkl.io (launched today)?