For more background, Flask embodies a lot of what Armin likes about how to build things, and there's room to disagree with him on it.
(for example, Flask attaches the request to a magic thread-local variable, while Django -- to pick the one I prefer -- requires you to explicitly pass it around and write functions to take it as an argument; there's a parallel in Armin's complaints about asyncio requiring you to explicitly pass things around instead of accessing magic thread-local storage)
From my understanding Armin's complaint isn't solely that you have to pass it around but the fragmentation of not having a standard way of doing it:
>> pass the event loop to all coroutines. That appears to be what a part of the community is doing. Giving a coroutine knowledge about what loop is going to schedule it makes it possible for the coroutine to learn about its task.
>> alternatively you require that the loop is bound to the thread. That also lets a coroutine learn about that. Ideally support both. Sadly the community is already torn of what to do.
Personally I'm not clear on a lot of the minority-use cases that spawns some of this complexity (e.g. when and why would you need to move event loops across threads?) and am quite out of my depth as well.
So I'm not the only one. I wrote larger-ish project while using asyncio and it is pain. The syntax is very unfamiliar (especially in 3.5 with async/await), the documentation is confusing and it's very hard to debug it in general. Also it's very hard to combine/stack multiple IO heavy events (make 5 calls to these URLs and whichever is done and returns these task run them in parallel).
Having worked with it a bit more I'd say that these things will work out as time goes on. Currently there is quite a bit of fragmentation of async-solutions in Python that are often somewhat-but-not-fully compatible; the docs need a lot of love as well. While the reference in itself is okay, cross-references and conceptual docs (the latter being extremely important IMHO) are lackluster or non-existent.
Also there seems to be a lot of confusion caused by the lack of docs around the difference of the keywords and the "asyncio" module (and it's various other forms). The former is just a coroutine / suspension engine and has relatively little to do with asyncio.
It is inherently somewhat limited by requiring the same underlying loop. But this is hard to change on a principal level...
---
In a broader sense, I like the increased visibility this gives to async APIs; with the "old ways" this could be easily overlooked. Since good API design is even more important in an asynch piece of software than in a sync piece of software I feel like this is quite an advantage, avoiding bugs and highlighting API issues directly.
With the old way, for example, it was relatively easy to confuse a synchronous method and a coroutine, since it's completely silent. You'd only notice this when stuff doesn't happen that was supposed to happen, and you can only see the bug at the call site if you know whether it's a method or a coroutine. This can't happen with the new keywords anymore. I think this is arguably their greatest advantage.
I agree that this was confusing in the docs. Docs can be improved. It really helped that this isn't my first crawler written using cooperative multitasking.
I had a similar experience with python 3's asyncio. I have worked with gevent, which has an arguably less "elegant" interface (with monkey patching, for instance), but with which it is so much easier to be productive. I was frustated for having dificulties understanding and using asyncio. The author is a better python programmer than I am, so I supposed there is really a problem there.
People don't write articles named "I don't understand gevent" because nobody expects to understand gevent.
Armin's complaints are for library writers. You will do much better using an async framework with support from the core language (asyncio) than a monkeypatch.
I think I've made the point a few times that I don't like this style of programming at all, because the coroutine layer turns into an Inner Platform [1] replicating all the control-flow structures the original language has, which then has to integrate with the original language which causes more than twice the complexity to emerge. But it's hard to bash together an example of how that happens in a comment, when it's all faked up and easy to dismiss. This is a great example of the complexity that can emerge. Some of that is incidental and will be fixed in future releases, but some of that is pretty fundamental, such as the initial list of all the sorts of things you need to learn about to use asyncio. Languages like Erlang or Go that have the event loop simply embedded into the language from day one have a much shorter list of such things you have to know, and it does all the things that this complicated list of asyncio concepts do.
But I will also admit some of this complexity is definitely Python-specific. I've been using Python since 1.5.2 was still the version you'd be most likely to encounter and I've liked it as a language for a while, but one release at a time, the language has been getting more and more complicated. By the time asyncio came around, the language was quite complicated, and sticking asyncio on top while integrating it with everything else is really a mess. Python has too darned many features at this point. I don't know exactly when it jumped the shark feature-wise, because individually they all make sense, but the sum total has gotten quite unwieldy. Watching new programmers try to learn Python, a language I used to suggest to such people as a good language to start with, has been a bit dispiriting lately.
I don't think the number of features is too problematic if it is properly gated. Python having 'http.server' or a Fractions module or tkinter or a matrix multiplication operator comes off as a pleasant surprise, it doesn't hamper the onboarding experience.
However, async, typing syntax, multiple string formatting syntaxes, and so on all do. The ship is being helmed irresponsibly in this regard. Letting decorator and keyword asyncio coexist, as well as threads and multiprocess, just feels like a huge mess.
Raw multi-threading was always a "blowing your feet of" feature in almost any way imaginable. That's true for Python as well and not only for use together with asyncio (but in this context even more so, I agree).
No one should write production MT code based on pthread-esque abstractions in any language without knowing very very well what they're doing.
I think you're right. I have python threads and processes meshed together in production code, and it's not pretty to look at. Sometimes it would crash for no reason, like this morning, when a job got stuck because one of its child process refused to die properly, thus blocking the join.
Due to the GIL and resulting coarse-grained nature of Python threading, it was very easy to write Python code that accidentally worked most of the time.
Interpreter instructions in CPython are atomic (due to the GIL, actually) and calling into extension functions is a single instruction (incidentally). This makes it impossible to corrupt built-in data structures, even if no locks are used. In some simple cases this is sufficient, esp. if no one's going to run it with !CPython...
This is also neat for other reasons (think sandboxing).
I agree that the features are not a problem. A large standard library and lots of packages are actually very welcome. The problem is the very design of the language, from the beginning.
I inherited a couple of Python 2.7 projects recently and after many years of toying with it I'm finally using Python seriously. Well, the syntax of the language is a mix of strange decisions. Part functional, part object oriented, not well designed in any of those parts. The so old school looking special __functions__. The annoying syntax errors one gets when refactoring and moving code around courtesy of the useless : and significant spacing. The even more annoying indentation errors when copy pasting code from the editor to the interactive interpreter. ["a", "b"].join(".") and ".".split("a.b") or other permutations, I still can't remember which one is correct. It's not a very inviting home to live in, a kind of non euclidean space.
It's strange that Python got where it is today. Maybe it's because among the languages that were big 10 years ago it's more immediate to work with than Java an C++, because PHP is despised by many professionals especially pre v7 when it was really ugly, because nobody bothered to build and maintain a numpy for Ruby and because JavaScript become big only recently. I feel that Python is good enough to do almost everything but not particularly nice to look at.
However I want to end on a lighter note quoting Matz. "No language can be perfect for everyone. I tried to make Ruby perfect for me, but maybe it's not perfect for you. The perfect language for Guido van Rossum is probably Python."
I feel that Python is more readable than most languages, and nice to look at it. I also like the magic functions. And whitespace never bothered me a second. I have the same amount of whitespace errors as I do forgetting a semicolon in PHP, which is infrequently. I don't know why you would consider JavaScript a better option with all its warts.
But you are right about the Python ecosystem. It is massive in the scientific and statistical computing. The other scripting languages outside of R aren't in the same ballpark in that domain.
> why you would consider JavaScript a better option with all its warts
I don't. Until recently there were so many ways to define a class that I was constantly forgetting how I did it last time. This is an order of magnitude worse than the split join thing. And all the wierd things we had to do to work around the callback hell. And the verbosity of function everywhere.
At least it is moving into the right direction with the last iterations of the language, which are mitigating those problems.
Ruby is not perfect but feels more logical and consistent.
"a.b".split(".") and ["a", "b"].join(".")
Ruby is bad if you want to do some functional programming but I find this logical: there are functional languages for that, which in turn are bad at object orientation. That's fine.
What I don't understand is Python doing OO by making us declare self in the method definitions as if it were a functional language that must explicitly carry around the state. Every other OO language knows how to handle self (JS is following a different OO model.) Python object orientation looks very low level. I was passing around self in C (no ++) to simulate OO: the pointer to the struct with the object data, function pointers and parent classes. Let's say that Python is very close to its implementation in C, but why?
In Python (almost?) any unqualified identifier you see in an expression is either a builtin function or it's defined/imported somewhere else in the file. I find Ruby a little stressful by comparison (without even getting into the awful cultural approval of defining the names of things procedurally, ensuring you'll never find where they came from...)
The lack of parens on function calls also adds uncertainty for me. I know in Python you can overload `__getattr__` and introduce just as much magic, but for the most part I can be confident that `a.b` doesn't do anything too crazy. That's the general trend for me -- Python is almost relentlessly boring, with a few little surprises that stick out mostly because everything else is so plain and sensible. Ruby is just a little crazier everywhere, partly because the language is a bit more eccentric and partly because the people who use it are all Ruby programmers :-)
My why was about why one should design a language in that way and not think harder and make it look better, but in the early days of Python there weren't many other OO languages around, so I can understand that it could have been natural to go somewhat low level and mimic C (even with all those __s). Maybe some language went more high level and died because of that. Python passed the test of time so that (self): probably was a good idea at the end of the 80s / early 90s.
In the case of Ruby, you can't name a function (which is a method) without executing it. That's why () don't matter much. The optional () also make Ruby a good language to write DSLs. By the way, if you want to get a reference to a method, you must prepend it with a &, pretty much like in C. Ha! :-) This demonstrates that every language has its quirks. Or you can call a method by sending a message to its object like object.send(:method) using a symbol named after the method. That's more or less a reference to it, which can be metaprogrammed because symbols can be built from strings ("something".to_sym). Is that the "defining the names of things procedurally" you don't like? On the other side, I find stressful that in Python you have to enumerate all your imports, like in Java. It's the same in Ruby, but I'm almost always programming in Rails and it auto imports everything. All those imports in Django and Web2py are tiresome. I got naming clashes with Rails only a couple of times in 10 years but I missed imports many times in Django yesterday.
I learned Python after Ruby. My trajectory was BASIC in the 80s, Pascal, C, Perl, a little TCL, Java and JavaScript since their beginning, Ruby since Rails, a little Python and PHP a few years later, much less Perl now and almost nothing of what preceded it, some Elixir. I keep using Ruby and JS, I'm using Python now. I insist that compared to Pascal, Java and Ruby Python looks illogical and unnecessarily complex, but I can understand why people with a different history can feel like Ruby is eccentric. I remember when I demoed it to some PHP developer many years ago, he said it was like writing in English, which was surprising because it is not how it looks to me, but it felt flattering.
I like the __'s, when listing methods you can immediately scan it and ignore all the built-in stuff if you want, to see what is special about this object.
In order to get "split" in ruby for a sequence, at least a one time, hopefully cleaned up by now, you end up mixing in some huge number of methods and made any method list in the console impossible to read.
Seems like a minor point to me, really. Sure, join could've been a member function of the list class, but that would prevent applying it to arbitrary iterables, no? In other words, delimiter.join(items) is more general than items.join(delimiter), because in the latter join must be a member function of the class of items or its ancestor class, and you won't be able to apply it to other iterable objects.
I haven't had much interaction with Ruby, but from my limited experience the syntax felt strictly less intuitive than Python. The only other languages where syntax felt more intuitive to me than Python is the ML family (with derivatives like OCaml, etc).
All the points raised here are minor and, mostly, subjective. Inability to comprehend this is the only reason such discussions are being repeated over and over again. It's utterly useless to discuss what syntax feels "natural" to whom - it's entirely dependent on what other languages (types of syntax, really) you already know.
There are some characteristics of syntax that we can discuss, for example, how large it is or what characters it tends to mainly use, but discussing these is apparently less fun than saying that something "feels illogical to me".
> Also, doesn't seem like you read the actual argument following the first sentence :)
Yeah, I was mainly referring to the @pmontra comments, like this one: "I find stressful that in Python you have to enumerate all your imports" and similar.
> Sure, join could've been a member function of the list class, but that would prevent applying it to arbitrary iterables, no?
Not true if your language supports multimethods, see Dylan, Common Lisp and CLOS or Nim and Julia for examples. Also not true if you add the "join" method high enough in the class hierarchy: as an example, in Pharo Smalltalk you have the following hierarchy: ProtoObject -> Object -> Collection -> SequencableCollection -> ArrayedCollection -> String with the "reduce" method being declared on Collection class (reduce being the easiest way to implement "join").
So in short: no. There are many interesting languages which implement various interesting techniques which solve various problems (like the so-called "Expression problem"); it's good to know about them even if you're not going to use them all that much (or at all).
> delimiter.join(items) is more general than items.join(delimiter)
But then you loose the ability to join a collection with a separator not being a String or you need to implement join even higher in the hierarchy (on Object most probably).
> The only other languages where syntax felt more intuitive to me
Yeah, this is what I'm campaigning against. This notion of "intuitiveness" is completely useless and is dependent on how your intuition was formed. All syntaxes of programming languages are artificial and man-made - there is nothing "natural" about them at all. In other words, they are all similarly alien and only get "intuitive" with practice. Programmers usually learn only a single syntax flavor during their careers, which is why they don't realize that the "intuitiveness" is just a function of familiarity. Learning some of the other kinds of syntax is good because it lets you observe how your "intuition" is shifting and changing in the process.
Your annoyingly patronizing tone aside, I'll try to address what you're saying.
> ...the class hierarchy: as an example, in Pharo Smalltalk you have the following hierarchy: ProtoObject -> Object -> Collection -> SequencableCollection -> ArrayedCollection -> String
Seems ridiculously over-engineered to me, but whatever, let's keep going..
> with the "reduce" method being declared on Collection class (reduce being the easiest way to implement "join").
'reduce' and 'join' are very different things. one is a generic function (aka fold, also exists in python as 'reduce'), the other is a string concatenation method that takes an iterable and produces a string. the latter can be implemented via the former, but they're not the same thing. no one's stopping you from using 'reduce' in Python instead of the built-in string member function 'join', btw.
> So in short: no. There are many interesting languages which implement various interesting techniques which solve various problems
Whatever you prefer the syntax for join and split to be, I prefer the opposite, but even if you quote Matz that doesn't mean your way is the right one.
I don't think they were quoting Matz as justification for why Python is wrong, they were quoting Matz to justify why what they don't like about Python isn't the wrong way.
The whole point is that every language has something that someone doesn't like about it and every language is "good enough" for some subset of people. Instead of attacking what points everyone does or does not like about X, it's instead better to learn from each other and take the best parts.
> It's not a very inviting home to live in, a kind of non euclidean space.
The surface of the earth is my favorite place to live on, but maybe you prefer infinite planes?
(Earth's surface is non euclidean, parallel lines intersect. And, yeah, I know, we don't live in the plane. For a more accurate example, this whole universe is non-euclidean, see space distortions around black holes.)
It has those. It as the greenlet library and used by eventlet and gevent. Many use those, and they've been there for many years. But Guido and others decided that is not acceptable. So we've gone the Twisted route (because everyone know Twisted is easy and fun) and now we have yields, co-routines, futures, awaits and all the other mess.
Lots of conversation between Armin and a few others about these approaches in reddit's /r/python thread for this article. I happen to think that gevent got a ton right, and the consensus seems to be that this 'new' approach is sort of half-baked, at least vis-a-vis python implementation.
Totally agree. Its a damn shame, too, it fit so elegantly into a programming style that looked very analgous to the threading library. Look at Tornado's StackContext to see the kind of mess that callpack passing creates. Its a gc nightmare.
Libmill runs in following environments:
Microarchitecture: x86_64, ARM
Compiler: gcc, clang
Operating system: Linux, OSX, FreeBSD, OpenBSD, NetBSD, DragonFlyBSD
Whether it works in different environments is not known - please, do report any successes or failures to the project mailing list.
It only works with gcc and clang now, no longer with windows, and is sprinkled with asm()s. That doesn't fly with the CPython is portable. I don't know how these go-like features could be created with ANSI C, but I'd love to be proven wrong.
Although I am not aware of any recent development, you should look at Stackless Python which was inspired by Limbo. Stackless Python has lightweight threads (stacklets) and synchronous channels a la various Bell labs family of languages (include Go). The greenlets that Gevent uses comes from Stackless.
Although I have used both CPython and Stackless.py, the specific differences are fading from my memory. While working on a Go style select() for stackless.py I recall there were minor differences in how the runnable list is represented and accessed. However this is really an implementation detail. For the most part, they are pretty much the same.
The biggest difference I remember in the current stackless.py implementation, the move to "continuelets" resulted in the inability to pickle "complex" stackless tasklets. So it becomes more difficult to stop a tasklet in one thread and restart it in another. Maybe the ability to control the recursion depth (including getting rid of it) may also be gone in stackless.py.
This is what happens when coders are rushing to code without real understanding what they are doing and why. What is worse - they borrow "features" without understanding from amateur JavaScript projects or C#.
What a bloated mess. This is clearly the second system syndrome, described in the Mystical Man-month.
In good old times futures were macros on top of delay and force special forms, and explicit message passing a-la Erlang would do the job.
What a confused comment. You're saying its bad because it's borrowed hastily from other languages, and a better solution is... borrowing features from Erlang?
Not at all. Erlang's approach to concurrency has been well-researched and validated (Akka).
Async, await and friends are mere standardized kludges - popular syntactic sugar without clear semantics and real world connection (explicit message passing mimics how biological systems do self-regulation).
So called enterprise languages, especially C++ are full of similar stuff (kludges).
> popular syntactic sugar without clear semantics and real world connection
Uhh... the point of them is that its as close to the semantics of synchronous code as possible. That's the 'real world connection' - your single threaded code can become asynchonous with just a few keyword changes. Rather than "sendRequest()" you do "await sendRequest()".
C# does all this stuff far clearer and cleaner (and probably more performant).
If they had taken more ideas from C# (especially ExecutionContexts), a lot of his complaints would fade away. He actually explicitly calls this out towards the bottom of the essay.
Second System Syndrome would be if all of this had been jammed into the 3.0 release, eight years ago.
It's clear that the async story stumbled into a tarpit after that, but I would be very surprised if it wasn't straightened out eventually into a clean syntax & implementation.
Although it will undoubtedly take longer than anyone would like to deprecate the crufty bits.
The Fluent Python book has a nice set of chapters on coroutines, futures, and async.io. They present not the whole of what it does but one way to do it, which helps.
I just got started on Python's asyncio. It took me a long time to understand how to get things done, but after about a week of doing small example projects in the evenings, it finally 'clicked'. For the record, I never worked with gevent before, and have only done a small webserver with Tornado a long time ago, so I can't comment on how it compares to other solutions. One thing that made me wait for asyncio is the fact that, despite being experimental, it has a lot more of an "official" feeling to it.
I'm trying to stay away entirely of the Python 3.4 way with coroutine decorators, and am using only await and async in Python 3.5. The async code I wrote has to live in parallel with regular synchronous Python code in a large scientific code base, but migrating our custom database adapter to an asynchronous codebase without breaking old synchronous code was surprisingly easy.
Debugging is, in my opinion, a pain. Stacktraces can be extremely long and very hard to understand. The only profiler that seemed to give useful results was pprofile (in sampling mode). I also still don't fully understand why there's both Futures and Tasks - I probably didn't spend enough time understanding the difference, but that just means the author of this blog post is right. Mixing asyncio with threads and/or processes, however, is surprisingly easy and elegant.
I hope the Python developers will have the courage to break backward compatibility in the asyncio module, and will remove the old yield from and @coroutine way of doing things. That would probably help a lot in reducing confusion. There's still not a lot of information about asyncio when you google for it, so the amount of existing code and examples that that would invalidate would not be too high.
All in all, we are very happy with asyncio. We use it mainly to add concurrency to small sections of our code base. By default, all our code is synchronous, with some heavy I/O-bound functions exposing async versions, too. asyncio allows us to parallelise these sections without the use of a thread pool, and thanks to Futures/Tasks and queues, it's actually very easy to do this in a "streaming" fashion if the order of processing of the outcome of your concurrent tasks matters. Add to that the executors which allow you to run stuff in sub-processes when you're CPU-bound (instead of I/O), and it makes for a fairly solid tool.
> I also still don't fully understand why there's both Futures and Tasks - I probably didn't spend enough time understanding the difference, but that just means the author of this blog post is right.
Because they are essentially the same.
Task is just extending Future and adds extra functionality (like for example keeping track of tasks schedule in given event loop).
It is created when you call ensure_future() or loop.create_task(). You are not supposed to create it directly, so if you're wondering whether you should use Future or Task you should use Future.
Long-time Twisted user (and lover) here. Lots of what we build are fairly web-transaction data translators (http to http or http to db APIs). I find that async.io, by avoiding callbacks, allows us to create a more classical-looking code structure. It works for us.
That being said:
a) the Twisted environment is extremely robust and battle tested.
b) I know that after six months of writing async.io code I'm still just tipping my toe into the capabilities and have yet to fully wrap my head around the details. Would love to see one of those single-topic O'Reilly books take a deep dive into async.io details.
Twisted is still popular. Problem is that in 2.x the async ecosystems do not work together and many things might not exist everywhere. Twisted seems to have lost a bit of ground with newer things like Kafka etc.
By now, people expect libraries to have thorough documentation on the Web, not just an O'Reilly book.
The fact that the port of Twisted to Python 3 is slow-going, and far from complete, also gives the suggestion that there are corners of the code that developers don't even understand anymore.
I'm working on some Asyncio stuff right now. Asynchronous programming seems pretty natural to me, but other people do struggle to wrap their heads around somewhat confusing terminology: tasks, co-routines, awaitables, event loops. Underneath the terminology the theory is pretty simple.
And boy is it powerful. If you ever find yourself doing network requests in a loop (for url in list: requests.get(url)) then a small bit of refactoring and a sprinkling of asyncio will speed this up immensely.
But it's not just for network calls, you can `await` on threads and processes. It's a joy and I think it's one of the best things in Python right now.
Have you tried comparing the performance of asyncio based network requests versus multithreaded requests? And also compared the relative complexity of the code?
I have never used asyncio in Python, mainly because the very use case you described is solved with multitheading, but that doesn't mean it's solved best that way of course.
The problem with multi-threading is knowing how to structure your program correctly to avoid issues with concurrent data-structure accesses, etc -- it requires more careful coding. With asyncio, you don't have to think about these issues.
> Have you tried comparing the performance of asyncio based network requests versus multithreaded requests
Nope, but the performance of an individual request overhead isn't much of a data point. The advantage is that it scales to thousands of connections easily, there are no concurrency/threading problems, and you can mix and match protocols easily. None of that is easy with threading. Threads can also be quite expensive to start and manage.
That specific case is handled by threading, yes, but if you're making a webservice that makes requests to a bunch of endpoints during when processing a HTTP request and also sends output to IRC/Slack whilst simultaneously serving files over FTP and launching a bunch of external processes for good measure then asyncio has your back.
Python's multithreading is insanely inefficient, because of the Guido von Rossum Memorial Boat Anchor. Anything in Python can mess with the innards of anything else at any time, including stuff in other threads. (See "setattr()"). There's no such thing as thread-local data in Python. This implies locking on everything. CPython has one big lock, the infamous Global Interpreter Lock. Some other implementations have more fine-grained locks, but still spend too much time locking and unlocking things. One Python program can thus use at most one CPU, no matter how many threads it has.
This basic problem has led to a pile of workarounds. First was "multiprocessing", which is a way to call subprocesses in a reasonably convenient fashion. A subprocess has far more overhead than a thread; it has its own Python interpreter (some code may be shared, but the data isn't) and a copy of all the compiled Python code. Launching a subprocess is expensive. So it's not a good way to handle, say, 10,000 remote connections.
Now there's "asyncio", which is the descendant of "Twisted Python". That was mostly used as a way for one Python instance to service many low-traffic network connections. The new "asyncio" is apparently more general, but hammering it into the language seems to have created a mess.
After the Python 3.x debacle, which essentially forked the language, we don't need this.
> There's no such thing as thread-local data in Python.
There is. threading.local in all aspects is thread local data.
> Now there's "asyncio", which is the descendant of "Twisted Python". That was mostly used as a way for one Python instance to service many low-traffic network connections. The new "asyncio" is apparently more general, but hammering it into the language seems to have created a mess.
I think the mess was created before 3.5. Had the whole thing started out with the async keywords we might have been spared `yield from` which is a beast in itself and a lot of the hacky machinery for legacy coroutines. I do think however we can still undo that damage.
threading.local in all aspects is thread local data.
You can still pass data attached to threading.local to another thread. Another thread may be able to get at threading.local data with setattr(). There's no isolation, so all the locking is still needed.
This is a hard problem. There's real thread-local data in C and C++, but it's not safe. If you pass a pointer to something on the stack to another thread, the address is invalid and the thread will probably crash trying to access it. C++ tries to prevent you from creating a reference to the stack, but the protection isn't airtight. In Rust, the compiler knows what's thread-local, as a consequence of the ownership system. Go kind of punts; data can be shared between coroutines, but the memory allocation system is mostly thread-safe. Mostly. Go's dicts are not thread-safe, and there's an exploit involving slice descriptor race conditions.
On what are you basing your statement that multithreading in Python is insanely inefficient? Despite the fact that the GIL prevents multiple threads from running in parallel, using multiple threads can give you a huge boost if IO is your bottleneck.
I think that it's irresponsible to make a blanket statement like this, because there are many use-cases for multiple threads in Python. Sure, one of the obvious ones (parallel processing) doesn't work, but besides that threads can be extremely useful.
I'm also unclear on what you mean by "no such thing as thread-local data" when there is `threading.local()` that does exactly that.
Lastly, I don't think multiprocessing was created as a workaround for threading per-se. Rather it was a workaround for the global interpreter lock.
And in either case my view is concurrency is best done in a language where there is proper support for it, whether the model is threaded or processor based concurrency.
Not that I disagree with (m)any of the points you raise, but with due respect I think it's a bit off-topic from my comment.
Were it my choice, any time the need for concurrency comes up at my job I'd prefer to use a statically typed, compiled langauge like C++ or Java (or, once I've familiarized myself with Rust's implementations, that language), and this kind of discussion wouldn't even come up. I like python as a rapid-prototyping language. For the kinds of numerical computations and data-laden I/O bound work I do I find it sorely lacking, and consider it an unfortunate choice for production work.
```python
from gevent import monkey
monkey.patch_all()
for url in list:
gevent.spawn(lambda: requests.get(url))
```
I concern this a problem already solved by gevent(or Erlang process / goroutine). Actually, I didn't see a benefit introduced by asyncio.
Monkey patching seems scary but it works quite well in real projects. At least in my medium sized (50KLOC) game project.
Handling network requests is the textbook example for green threads (mostly because it fits in a paragraph), and if you never tried it you should step outside of the Python ecosystem for a short while to get a clear picture.
Tasks that require bare async programming are incredibly rare. There's a small set of patterns that covers almost every application ever written, but it's not completely covered by Twister (or at least didn't use to be, there's a few years that I don't touch it).
Honestly, there's no problem with Python exposing the low level async primitives. That's good, and very pythonic. My only problem is that everybody is talking like this is the complete package and Python is now a good choice for async programming.
I have used this under Linux, OSX and Windows. It's cool to add the Thread Count field in Task Manager and then see something I wrote use so many threads! I am more of a sys admin, so this code could be better - but it seems to work very well. :-)
Well, no, of course not. Don't be stupid. The default ThreadPoolExecutor size is the number of cores you have on your laptop. In my case, 8. So that's 8 concurrent connections at once. That's, frankly, shit.
Use asyncio (or roll your own toolkit, or just spawn 'nmap') and you get 20000 concurrent connections.
The connections are actually handled by the OS in parallel, it's the callbacks to Python that are not. Why do you need a whole OS thread to handle sending a SYN/ACK?
As you well know 20000 threads is not a great way to do anything. Especially in Python.
quick github tip: if you click the line number on the left you can get a link directly to the line you are referring to.
extra tip: after clicking the line, press 'y' on your keyboard and you'll get a link to the file in it's state at the current commit so future commits won't break your old hyperlinks.
He is the author of several popular projects including the web framework "Flask". This makes him a person very respected in the Python community - personally I love his taste for interface design.
I wish he could interact better with the core team, because some of his rants are not as constructive as they might be.
Simplicity is a core value of the Python community; this kind of change goes through a long discussion before getting implemented, so I wonder how we ended up with such convoluted design (the reasoning is documented in the PEP system but most of it is beyond my wizardry level).
Perhaps this is me showing some cultural bias (I'm Brazilian), but in my social context it is considered rude to make this kind of criticism in public unless you tried your best to address the problem in a more restricted exchange. The more authority you have, more you are expected to come with a better proposal instead of just pointing out the mess.
asyncio is not my construction site and I do not intend to make it mine. I am not an expert in anything asynchronous and as such will not try to make myself that person.
This entire adventure started because I was looking for a way to get logical call contexts working in it and this shows that there is not enough internal machinery to support that at the moment.
I'm not going to fight for that however. I hope my post was rude.
Thanks for taking the time to write out your concrete experiences and qualitative impressions in such detail. It's really a tremendous service and I'm sure it'll be well received.
For what it's worth I would like to do a PEP on accessing event loops from coroutines and logical call contexts but having read some of the conversations that already took place to improve asyncio I am less thrilled about being involved.
I suggest to proceed with the PEP if you have a good idea about the API. Get it out there, it's not required to spend hours on mailing lists - other people can do that if they like it.
> unless you tried your best to address the problem in a more restricted exchange.
That sounds like orders of magnitude more work. In case you are not interested in performing that work, it seems more constructive to me to share your thoughts rather than to say nothing.
Whoa... Armin has never, to my recollection, been malicious in his criticisms of Python 3. He simply disagrees with some of the core team's decisions. And he's far from the only one.
It's not Armin's duty to interact in any way with the core team. Hell, the core team should be working to please Armin if you ask me. He's your target user, and he's disappointed with your product, it's not his fault.
I think Armin does a wonderful job providing a voice for those of us who are increasingly disenchanted with Python 3.
I agree with Armin that Python is becoming burdensomly complicated. Since Python 3.2, the language has started to accumulate a lot of features, Guido seems to be saying yes to everything. Furthermore, the Python 3 fiasco has severely affected Python's popularity. It would be in much greater use now if the clean-up had been done more gradually.
I don't quite see why it's being downvoted. Like it or not, but that's what is happening: Python's 2.4-2.7 success was largely due to it's simplicity. It more or less did correspond to values defined in `import this`. It wasn't particularly performant nor "safe/foolproof", it wasn't even that powerful as a language. It lacked (and still lacks) good error reporting system. But it was lean, easy to get started with, agile enough, expressive enough. It was always possible to make something unintelligible using reflection and magic methods, but it is easily distinguishable form "how stuff should be written" and otherwise you could be quite sure there won't be any surprises.
This is why python is still used for it's purpose. CLI utils, bots/crawlers, tf/pandas/scikit-learn, REST-APIs on top of flask.
Then hard times came. First, many year long story (still not concluded) of transition between 2 and 3. Then all this stuff. Sure, there is such thing as progress. Stuff is invented for some purpose. But now, in 2016 and at v3.6 — Python isn't what it has been loved for. Not easy-to-start-with, nor simple. This return/yield fuck up, showed in the article is absolutely huge deal, for example, and it is not about asyncio per se. Async stuff is always complicated, it wouldn't be that bad if it was all it's about.
If some 5 years ago one would use Python just because "why not, I just need to get stuff done", now it's quite likely that after struggling with all this micro-nuisances he would go with golang/js/php/whatever instead.
The thing about Python is that unlike most other languages, you don't have to deal with that complexity. You can still write Python 2.7 style code, no problem.
This approach to async, though, is just a language feature that's becoming mainstream right now. C# has it, ES7 has it, C++ has a working paper on it etc. Python actually had the benefit of watching how things work out elsewhere before implementing it all.
The idea that complexity doesn't matter if you don't use it sets Python on the path to becoming the next C++. Pick the subset of the language that you like. Then carefully watch your libraries, because if they use a different subset, you still have to deal with what's in there (and Python is a library heavy language!). It goes against everything Python stands (stood) for. Definitely not what I'm looking for.
This is not at all a new problem for Python. In fact, it's possibly more of a problem for Python than it is for C++, because Python's dynamic nature and exposure of many internal mechanisms makes it possible to do some pretty crazy stuff in the libraries.
For example, speaking of async - even 2.7 already has Twisted, and an ecosystem of libraries around it.
The only two ways I can see it being solved is either by making it more of a toy language (which is great if you're just writing short scripts, but it's not really what it's supposed to be about); or by having a very centralized "best practices" enforcement that basically forces libraries to conform through peer pressure, like Java - which has its own disadvantages aplenty.
That's because it's not unneeded complexity. Making you stop and think about what you're doing when you say "just treat these bytes as a string" (or vice versa) is to prevent buggy code that only works properly with Latin-1, and similar kind of thing.
If some library messes it up, and requires you to provide bytes that are semantically a string, it's a design flaw in the library.
okay try and use any networking library such as zmq, and see what a pain in the butt it is, most definitely refuting the idea that "you can write python 2.7 code if you want". You're going to be polluting your code with .encode .decode all over the shop. Same with xrange, dict access. All sorts of cruft has been introduced in the pursuit of "seriousness". And now we have this post (and most of the comments) saying that 3's flagship win, asyncio, is actually a dog. I reluctantly moved to 3 last year and honestly, I cannot find any wins whatsoever. Seriously considering Lua, although it also has its 5.3 v 5.2 problems not to mention Luajit "fork".
> You're going to be polluting your code with .encode .decode all over the shop.
Well, do those encode/decode calls serve a purpose? Does encoding matter?
If so, they're not cruft - they do something that needs to be done. You may dislike the fact that it's extra complexity, but any speaker of a language for which ASCII (or Latin-1) is not sufficient will rejoice knowing that you can't write code that breaks when we try to use it anymore. I remember how much of a hassle this kind of stuff was back in DOS/Win9x days, and even in early 00s; and I also remember how a hard push for Unicode as the string encoding in mainstream languages like Java and C# did a lot to rectify that.
If not, then why would you need to encode something just to decode it later, or vice versa? Why not just pass bytes/bytearray around? Again, if the library requires doing so (because they demand the data to be passed as a str specifically, even though it's never actually treated as a string), then that's a bug in said library, and you should complain to/about its authors instead.
As If JS isn't becoming more complicated by the year, and PHP didn't add a ton of features in 5. Anyway, Python ranks in the top 4 most popular language rankings on multiple sites, and plenty of people are still learning it.
There are a few new concepts that need to be learned when beginning work with asyncio as well as some confusing naming choices (who would expect that a coroutine is not the same as a coroutine function?). Some of the tooling is also lacking. Armin points out, no doubt with a look at server frameworks, that there is no elegant way to access the context of a task.
That being said, as someone who started working with asyncio in Python 3.5 none of it feels particularly difficult to understand. Asyncio needs more work, sure, but the API so far is relatively straightforward.
The overloading of iterators/generators is a bit odd. But the same approach that was taken with JavaScript -- and it's nothing anyone working with Python 3.5 or above will be exposed to. If I recall correctly, Python 3.6 will even feature async generators. Any developers diving into this using that release won't have to ask themselves why they can't 'yield' from within an 'async' function.
I don't find it hard to write asyncio code but I find it hard to generically interface with other people's asyncio code and that is lacking basic support like a context object or logical call contexts or just the lack of agreed upon usage patterns.
It does not help that asyncio evolves in the stdlib and changes with every major Python version. It might be less of an issue if this was pip installable I suppose. Right now writing utility code for asyncio is targeting many things.
Definitely fair, as time progresses I think cleaner patterns will emerge. By time many of the major libraries were around, there was a semi-standard Python programming style that was fairly easy to adhere to. Since this is somewhat of a paradigm shift for the stdlib it will take a little time too.
Personally I have been looking at some of the patterns taken by the team on aiohttp - I think they've done a good job and, iirc, one of their members is a core contributor to Python.
I was talking with David Beazley and he had some of the same confusions around asyncio. I think it would be nice to revisit it with a round table and fix it in 3.7.
I wouldn't go so far as to say it's a language FOR beginners. But it is easier to understand some concepts in python than staring at god awful C or Perl or most other languages as a totally new person to programming.
> wasn't it supposed to be a language for beginners?
No. It was supposed to be a language in which programmers at widely different skill levels, from beginner to expert, could be productive. Easy to pick up the basics, but also easy to use more advanced techniques when you find you need them.
I didn't need to understand everything about asyncio to use it successfully. It's definitely a lifesaver when it comes to making lots of requests; hours of waiting turn into seconds.
Python core developer Brett Cannon wrote a nice(and long) post about understanding it a couple of months ago[1], that might help.
After reading that I felt like I could use it, but never got the chance to try.
At home-assistant.io we just migrated our core from being based on threads + locks to use asyncio. We managed to keep a full backwards compatible API available so we can slowly migrate other parts of the system over to async.
Asyncio has a steep initial learning curve (especially in our hybrid setup) but it's well worth it. We target low resource computers like Raspberry Pi and using async over threads has speed up things a lot.
The biggest catch is that while writing code you have to think about every function that you use. Is it doing I/O, is it a coroutine or is it callback/async friendly.
That's four opcodes. Only one operation will run at a time, but one thread might preempt the other midway through the four operations, and `a` will have been incremented by 1, not two.
In some sense, mulithreading with Python's GIL is the worst of both worlds: you can use only one CPU, but you still need to account for concurrent state mutation. (Of course, it doesn't have the function color problem of asyncio http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...)
Probably because you don't have to worry about context switches in the middle of operations, since they happen at 'await' or 'yield's in coroutines, rather than whenever the OS task timer interrupts
We no longer need locks in the core because there is now only 1 thread that interacts with the core. We still have other threads but all they do is schedule tasks to be run on the event loop.
There's still a possibility that a lock might be needed if you use await/yield from inside critical section (another co-routine could make changes at that point). If you don't yield then indeed locks are not needed.
I've been a gevent user for a long time and Python's decision to "bless" twisted by adopting it's patterns was a watershed moment for me, and basically was the beginning of the end of my belief that I'd ever adopt Python 3.
User jerf's comment that asyncio "more than [doubles] the complexity" is absolutely correct. Watch this video of Guido talking about tulip...or struggling to talk about tulip, rather. It's clear the dude is out of his depth and my god the recent changes to the language show that the inmates are now running the asylum... Seems like Python, in it's effort to chase the latest fads, is no longer the language I would endorse to someone new to programming. Whether you think that's a meaningful litmus test or not, the staggering amount of crap that's infiltrated the language now completely flies in the face of the zen of python's statement that there should be one and preferably only one way of doing things. Fuck. I'm going to go code some lua now.
Oh come on. Hyperbole much? You're acting as if it's impossible to write any sort of meaningful code in 3.x without embracing asyncio. A few 2to3-ish language changes excepted, you can punch the exact same code into the Python 3.x interpreter that you have been for the past 15 years.
I agree with coleifer (and others) that Python 3 is losing "the zen". The new f-string stuff is a perfect example: a bad, retrogressive idea that never should have been approved.
I have mixed feelings about f strings. Having the string literal bound to the variable name feels kind of backwards -- that one could break interpolated strings inadvertently by refactoring a variable name. On the other the str.format syntax is a bit verbose, even in shorthand notation.
That said, I use Python 3 every day for almost everything without need to use every shiny new feature.
It's Python we are talking about right? Refactoring a variable name in a non type safe language has always been prone to break something. If you trust your IDE to do refactoring on that level I would also trust it would rename inside the string.
> I've been a gevent user for a long time and Python's
> decision to "bless" twisted by adopting it's patterns
> was a watershed moment for me, and basically was the
> beginning of the end of my belief that I'd ever adopt
> Python 3.
This was exactly my experience as well. I had used twisted for quite a while, and never really liked it (aside: my god was inlineCallbacks a great improvement though!). Then later I started using gevent, and used it very successfully for a few years. So much more of a pleasant experience to use.
I still do quite a bit of work with python2. These days I am using Go in quite a few of my personal projects, and for work projects where I am given the choice.
I had the pleasure of working over IRC with you (at Ellington) when I worked for CMG Digital, and I found you to be one of the most knowledgeable Python developers I had met.
But...
I think you're blowing this out of proportion. I think that yes, the implementation details are far too complicated, and yes there has been some serious (IMHO) mistakes made (e.g., type hints, format strings, etc.) in Python 3, but overall the language is terrific and getting better.
On the other hand, it's hard to argue with the track record of success of Twisted's approach to async programming in Python. It was at the point where enough people were clamoring for something to be blessed that it makes sense to pick the one whose approaches and patterns are as stable and battle-tested as Twisted's.
What's missing from the asyncio docs, IMHO, is a section on common asyncio idioms and patterns. I understand the pieces individually, but I struggle to get a more global, Gestalt understanding of the system.
The one thing that I think is absolutely lovely is the syntax:
async def spam(eggs):
...
It can't get much better than that.
My hope is that the implementation will become:
A) simpler
B) more optimized
C) out of the box functional (e.g., no need to manage event loops and other things yourself)
D) unified (e.g., the zen of python even states that there should be one way to do something)
I also strongly agree with the comment @RodericDay made herein, pertaining the new typing syntax and the addition of yet another string formatting syntax (a dangerously unexplicit one).
Actually using asyncio tools is quite easy and straighforward. However, writting an asyncio lib, or god saves you, an asyncio framework, is really, really hard.
214 comments
[ 3.4 ms ] story [ 253 ms ] thread0. http://lucumr.pocoo.org/projects/
Btw. does someone know why some things where moved from Pocoo to Pallets?
[1]: https://www.palletsprojects.com/blog/hello/
(for example, Flask attaches the request to a magic thread-local variable, while Django -- to pick the one I prefer -- requires you to explicitly pass it around and write functions to take it as an argument; there's a parallel in Armin's complaints about asyncio requiring you to explicitly pass things around instead of accessing magic thread-local storage)
>> pass the event loop to all coroutines. That appears to be what a part of the community is doing. Giving a coroutine knowledge about what loop is going to schedule it makes it possible for the coroutine to learn about its task.
>> alternatively you require that the loop is bound to the thread. That also lets a coroutine learn about that. Ideally support both. Sadly the community is already torn of what to do.
Personally I'm not clear on a lot of the minority-use cases that spawns some of this complexity (e.g. when and why would you need to move event loops across threads?) and am quite out of my depth as well.
Having worked with it a bit more I'd say that these things will work out as time goes on. Currently there is quite a bit of fragmentation of async-solutions in Python that are often somewhat-but-not-fully compatible; the docs need a lot of love as well. While the reference in itself is okay, cross-references and conceptual docs (the latter being extremely important IMHO) are lackluster or non-existent.
Also there seems to be a lot of confusion caused by the lack of docs around the difference of the keywords and the "asyncio" module (and it's various other forms). The former is just a coroutine / suspension engine and has relatively little to do with asyncio.
---
Specifically your last request: https://docs.python.org/3/library/asyncio-task.html#asyncio....
It is inherently somewhat limited by requiring the same underlying loop. But this is hard to change on a principal level...
---
In a broader sense, I like the increased visibility this gives to async APIs; with the "old ways" this could be easily overlooked. Since good API design is even more important in an asynch piece of software than in a sync piece of software I feel like this is quite an advantage, avoiding bugs and highlighting API issues directly.
With the old way, for example, it was relatively easy to confuse a synchronous method and a coroutine, since it's completely silent. You'd only notice this when stuff doesn't happen that was supposed to happen, and you can only see the bug at the call site if you know whether it's a method or a coroutine. This can't happen with the new keywords anymore. I think this is arguably their greatest advantage.
This is the part that sends work to a thread: https://github.com/cocrawler/cocrawler/blob/master/cocrawler...
I agree that this was confusing in the docs. Docs can be improved. It really helped that this isn't my first crawler written using cooperative multitasking.
Armin's complaints are for library writers. You will do much better using an async framework with support from the core language (asyncio) than a monkeypatch.
But I will also admit some of this complexity is definitely Python-specific. I've been using Python since 1.5.2 was still the version you'd be most likely to encounter and I've liked it as a language for a while, but one release at a time, the language has been getting more and more complicated. By the time asyncio came around, the language was quite complicated, and sticking asyncio on top while integrating it with everything else is really a mess. Python has too darned many features at this point. I don't know exactly when it jumped the shark feature-wise, because individually they all make sense, but the sum total has gotten quite unwieldy. Watching new programmers try to learn Python, a language I used to suggest to such people as a good language to start with, has been a bit dispiriting lately.
[1]: https://en.wikipedia.org/wiki/Inner-platform_effect
However, async, typing syntax, multiple string formatting syntaxes, and so on all do. The ship is being helmed irresponsibly in this regard. Letting decorator and keyword asyncio coexist, as well as threads and multiprocess, just feels like a huge mess.
That's the main issue. You can work with asyncio and threads at the same time but you need to know very well how the frameworks work.
No one should write production MT code based on pthread-esque abstractions in any language without knowing very very well what they're doing.
This is also neat for other reasons (think sandboxing).
I inherited a couple of Python 2.7 projects recently and after many years of toying with it I'm finally using Python seriously. Well, the syntax of the language is a mix of strange decisions. Part functional, part object oriented, not well designed in any of those parts. The so old school looking special __functions__. The annoying syntax errors one gets when refactoring and moving code around courtesy of the useless : and significant spacing. The even more annoying indentation errors when copy pasting code from the editor to the interactive interpreter. ["a", "b"].join(".") and ".".split("a.b") or other permutations, I still can't remember which one is correct. It's not a very inviting home to live in, a kind of non euclidean space.
It's strange that Python got where it is today. Maybe it's because among the languages that were big 10 years ago it's more immediate to work with than Java an C++, because PHP is despised by many professionals especially pre v7 when it was really ugly, because nobody bothered to build and maintain a numpy for Ruby and because JavaScript become big only recently. I feel that Python is good enough to do almost everything but not particularly nice to look at.
However I want to end on a lighter note quoting Matz. "No language can be perfect for everyone. I tried to make Ruby perfect for me, but maybe it's not perfect for you. The perfect language for Guido van Rossum is probably Python."
But you are right about the Python ecosystem. It is massive in the scientific and statistical computing. The other scripting languages outside of R aren't in the same ballpark in that domain.
I don't. Until recently there were so many ways to define a class that I was constantly forgetting how I did it last time. This is an order of magnitude worse than the split join thing. And all the wierd things we had to do to work around the callback hell. And the verbosity of function everywhere.
At least it is moving into the right direction with the last iterations of the language, which are mitigating those problems.
Hmm, i've always felt the exact opposite, that Python feels just right syntax-wise (this after years of C/C++, Perl, bash, JavaScript, etc).
What would be your example of a language residing in "euclidean space"?
"a.b".split(".") and ["a", "b"].join(".")
Ruby is bad if you want to do some functional programming but I find this logical: there are functional languages for that, which in turn are bad at object orientation. That's fine.
What I don't understand is Python doing OO by making us declare self in the method definitions as if it were a functional language that must explicitly carry around the state. Every other OO language knows how to handle self (JS is following a different OO model.) Python object orientation looks very low level. I was passing around self in C (no ++) to simulate OO: the pointer to the struct with the object data, function pointers and parent classes. Let's say that Python is very close to its implementation in C, but why?
In Python (almost?) any unqualified identifier you see in an expression is either a builtin function or it's defined/imported somewhere else in the file. I find Ruby a little stressful by comparison (without even getting into the awful cultural approval of defining the names of things procedurally, ensuring you'll never find where they came from...)
The lack of parens on function calls also adds uncertainty for me. I know in Python you can overload `__getattr__` and introduce just as much magic, but for the most part I can be confident that `a.b` doesn't do anything too crazy. That's the general trend for me -- Python is almost relentlessly boring, with a few little surprises that stick out mostly because everything else is so plain and sensible. Ruby is just a little crazier everywhere, partly because the language is a bit more eccentric and partly because the people who use it are all Ruby programmers :-)
In the case of Ruby, you can't name a function (which is a method) without executing it. That's why () don't matter much. The optional () also make Ruby a good language to write DSLs. By the way, if you want to get a reference to a method, you must prepend it with a &, pretty much like in C. Ha! :-) This demonstrates that every language has its quirks. Or you can call a method by sending a message to its object like object.send(:method) using a symbol named after the method. That's more or less a reference to it, which can be metaprogrammed because symbols can be built from strings ("something".to_sym). Is that the "defining the names of things procedurally" you don't like? On the other side, I find stressful that in Python you have to enumerate all your imports, like in Java. It's the same in Ruby, but I'm almost always programming in Rails and it auto imports everything. All those imports in Django and Web2py are tiresome. I got naming clashes with Rails only a couple of times in 10 years but I missed imports many times in Django yesterday.
I learned Python after Ruby. My trajectory was BASIC in the 80s, Pascal, C, Perl, a little TCL, Java and JavaScript since their beginning, Ruby since Rails, a little Python and PHP a few years later, much less Perl now and almost nothing of what preceded it, some Elixir. I keep using Ruby and JS, I'm using Python now. I insist that compared to Pascal, Java and Ruby Python looks illogical and unnecessarily complex, but I can understand why people with a different history can feel like Ruby is eccentric. I remember when I demoed it to some PHP developer many years ago, he said it was like writing in English, which was surprising because it is not how it looks to me, but it felt flattering.
In order to get "split" in ruby for a sequence, at least a one time, hopefully cleaned up by now, you end up mixing in some huge number of methods and made any method list in the console impossible to read.
Seems like a minor point to me, really. Sure, join could've been a member function of the list class, but that would prevent applying it to arbitrary iterables, no? In other words, delimiter.join(items) is more general than items.join(delimiter), because in the latter join must be a member function of the class of items or its ancestor class, and you won't be able to apply it to other iterable objects.
I haven't had much interaction with Ruby, but from my limited experience the syntax felt strictly less intuitive than Python. The only other languages where syntax felt more intuitive to me than Python is the ML family (with derivatives like OCaml, etc).
All the points raised here are minor and, mostly, subjective. Inability to comprehend this is the only reason such discussions are being repeated over and over again. It's utterly useless to discuss what syntax feels "natural" to whom - it's entirely dependent on what other languages (types of syntax, really) you already know.
There are some characteristics of syntax that we can discuss, for example, how large it is or what characters it tends to mainly use, but discussing these is apparently less fun than saying that something "feels illogical to me".
Also, doesn't seem like you read the actual argument following the first sentence :)
Yeah, I was mainly referring to the @pmontra comments, like this one: "I find stressful that in Python you have to enumerate all your imports" and similar.
> Sure, join could've been a member function of the list class, but that would prevent applying it to arbitrary iterables, no?
Not true if your language supports multimethods, see Dylan, Common Lisp and CLOS or Nim and Julia for examples. Also not true if you add the "join" method high enough in the class hierarchy: as an example, in Pharo Smalltalk you have the following hierarchy: ProtoObject -> Object -> Collection -> SequencableCollection -> ArrayedCollection -> String with the "reduce" method being declared on Collection class (reduce being the easiest way to implement "join").
So in short: no. There are many interesting languages which implement various interesting techniques which solve various problems (like the so-called "Expression problem"); it's good to know about them even if you're not going to use them all that much (or at all).
> delimiter.join(items) is more general than items.join(delimiter)
But then you loose the ability to join a collection with a separator not being a String or you need to implement join even higher in the hierarchy (on Object most probably).
> The only other languages where syntax felt more intuitive to me
Yeah, this is what I'm campaigning against. This notion of "intuitiveness" is completely useless and is dependent on how your intuition was formed. All syntaxes of programming languages are artificial and man-made - there is nothing "natural" about them at all. In other words, they are all similarly alien and only get "intuitive" with practice. Programmers usually learn only a single syntax flavor during their careers, which is why they don't realize that the "intuitiveness" is just a function of familiarity. Learning some of the other kinds of syntax is good because it lets you observe how your "intuition" is shifting and changing in the process.
> ...the class hierarchy: as an example, in Pharo Smalltalk you have the following hierarchy: ProtoObject -> Object -> Collection -> SequencableCollection -> ArrayedCollection -> String
Seems ridiculously over-engineered to me, but whatever, let's keep going..
> with the "reduce" method being declared on Collection class (reduce being the easiest way to implement "join").
'reduce' and 'join' are very different things. one is a generic function (aka fold, also exists in python as 'reduce'), the other is a string concatenation method that takes an iterable and produces a string. the latter can be implemented via the former, but they're not the same thing. no one's stopping you from using 'reduce' in Python instead of the built-in string member function 'join', btw.
> So in short: no. There are many interesting languages which implement various interesting techniques which solve various problems
Ugh, i give up :)
The whole point is that every language has something that someone doesn't like about it and every language is "good enough" for some subset of people. Instead of attacking what points everyone does or does not like about X, it's instead better to learn from each other and take the best parts.
The surface of the earth is my favorite place to live on, but maybe you prefer infinite planes?
(Earth's surface is non euclidean, parallel lines intersect. And, yeah, I know, we don't live in the plane. For a more accurate example, this whole universe is non-euclidean, see space distortions around black holes.)
If it's doable in C, why not in Python?
https://www.reddit.com/r/Python/comments/5a6gmv/i_dont_under...
https://github.com/sustrik/libmill/blob/master/libmill.h
It only works with gcc and clang now, no longer with windows, and is sprinkled with asm()s. That doesn't fly with the CPython is portable. I don't know how these go-like features could be created with ANSI C, but I'd love to be proven wrong.
[1] http://www.stackless.com/
[2] http://doc.pypy.org/en/latest/stackless.html
The biggest difference I remember in the current stackless.py implementation, the move to "continuelets" resulted in the inability to pickle "complex" stackless tasklets. So it becomes more difficult to stop a tasklet in one thread and restart it in another. Maybe the ability to control the recursion depth (including getting rid of it) may also be gone in stackless.py.
What a bloated mess. This is clearly the second system syndrome, described in the Mystical Man-month.
In good old times futures were macros on top of delay and force special forms, and explicit message passing a-la Erlang would do the job.
Async, await and friends are mere standardized kludges - popular syntactic sugar without clear semantics and real world connection (explicit message passing mimics how biological systems do self-regulation).
So called enterprise languages, especially C++ are full of similar stuff (kludges).
Uhh... the point of them is that its as close to the semantics of synchronous code as possible. That's the 'real world connection' - your single threaded code can become asynchonous with just a few keyword changes. Rather than "sendRequest()" you do "await sendRequest()".
If they had taken more ideas from C# (especially ExecutionContexts), a lot of his complaints would fade away. He actually explicitly calls this out towards the bottom of the essay.
It's clear that the async story stumbled into a tarpit after that, but I would be very surprised if it wasn't straightened out eventually into a clean syntax & implementation.
Although it will undoubtedly take longer than anyone would like to deprecate the crufty bits.
Oh, and it is mythical, not mystical.
I'm trying to stay away entirely of the Python 3.4 way with coroutine decorators, and am using only await and async in Python 3.5. The async code I wrote has to live in parallel with regular synchronous Python code in a large scientific code base, but migrating our custom database adapter to an asynchronous codebase without breaking old synchronous code was surprisingly easy.
Debugging is, in my opinion, a pain. Stacktraces can be extremely long and very hard to understand. The only profiler that seemed to give useful results was pprofile (in sampling mode). I also still don't fully understand why there's both Futures and Tasks - I probably didn't spend enough time understanding the difference, but that just means the author of this blog post is right. Mixing asyncio with threads and/or processes, however, is surprisingly easy and elegant.
I hope the Python developers will have the courage to break backward compatibility in the asyncio module, and will remove the old yield from and @coroutine way of doing things. That would probably help a lot in reducing confusion. There's still not a lot of information about asyncio when you google for it, so the amount of existing code and examples that that would invalidate would not be too high.
All in all, we are very happy with asyncio. We use it mainly to add concurrency to small sections of our code base. By default, all our code is synchronous, with some heavy I/O-bound functions exposing async versions, too. asyncio allows us to parallelise these sections without the use of a thread pool, and thanks to Futures/Tasks and queues, it's actually very easy to do this in a "streaming" fashion if the order of processing of the outcome of your concurrent tasks matters. Add to that the executors which allow you to run stuff in sub-processes when you're CPU-bound (instead of I/O), and it makes for a fairly solid tool.
Because they are essentially the same.
Task is just extending Future and adds extra functionality (like for example keeping track of tasks schedule in given event loop).
It is created when you call ensure_future() or loop.create_task(). You are not supposed to create it directly, so if you're wondering whether you should use Future or Task you should use Future.
Would you happen to have some code examples, or just examples of what kind of sample projects you built?
That being said: a) the Twisted environment is extremely robust and battle tested. b) I know that after six months of writing async.io code I'm still just tipping my toe into the capabilities and have yet to fully wrap my head around the details. Would love to see one of those single-topic O'Reilly books take a deep dive into async.io details.
Can you explain why this is? What's the central issue?
Did something happen that it's now fallen out of favor?
The fact that the port of Twisted to Python 3 is slow-going, and far from complete, also gives the suggestion that there are corners of the code that developers don't even understand anymore.
And boy is it powerful. If you ever find yourself doing network requests in a loop (for url in list: requests.get(url)) then a small bit of refactoring and a sprinkling of asyncio will speed this up immensely.
But it's not just for network calls, you can `await` on threads and processes. It's a joy and I think it's one of the best things in Python right now.
I have never used asyncio in Python, mainly because the very use case you described is solved with multitheading, but that doesn't mean it's solved best that way of course.
Nope, but the performance of an individual request overhead isn't much of a data point. The advantage is that it scales to thousands of connections easily, there are no concurrency/threading problems, and you can mix and match protocols easily. None of that is easy with threading. Threads can also be quite expensive to start and manage.
That specific case is handled by threading, yes, but if you're making a webservice that makes requests to a bunch of endpoints during when processing a HTTP request and also sends output to IRC/Slack whilst simultaneously serving files over FTP and launching a bunch of external processes for good measure then asyncio has your back.
This basic problem has led to a pile of workarounds. First was "multiprocessing", which is a way to call subprocesses in a reasonably convenient fashion. A subprocess has far more overhead than a thread; it has its own Python interpreter (some code may be shared, but the data isn't) and a copy of all the compiled Python code. Launching a subprocess is expensive. So it's not a good way to handle, say, 10,000 remote connections.
Now there's "asyncio", which is the descendant of "Twisted Python". That was mostly used as a way for one Python instance to service many low-traffic network connections. The new "asyncio" is apparently more general, but hammering it into the language seems to have created a mess.
After the Python 3.x debacle, which essentially forked the language, we don't need this.
There is. threading.local in all aspects is thread local data.
> Now there's "asyncio", which is the descendant of "Twisted Python". That was mostly used as a way for one Python instance to service many low-traffic network connections. The new "asyncio" is apparently more general, but hammering it into the language seems to have created a mess.
I think the mess was created before 3.5. Had the whole thing started out with the async keywords we might have been spared `yield from` which is a beast in itself and a lot of the hacky machinery for legacy coroutines. I do think however we can still undo that damage.
You can still pass data attached to threading.local to another thread. Another thread may be able to get at threading.local data with setattr(). There's no isolation, so all the locking is still needed.
This is a hard problem. There's real thread-local data in C and C++, but it's not safe. If you pass a pointer to something on the stack to another thread, the address is invalid and the thread will probably crash trying to access it. C++ tries to prevent you from creating a reference to the stack, but the protection isn't airtight. In Rust, the compiler knows what's thread-local, as a consequence of the ownership system. Go kind of punts; data can be shared between coroutines, but the memory allocation system is mostly thread-safe. Mostly. Go's dicts are not thread-safe, and there's an exploit involving slice descriptor race conditions.
You can in most languages. Only rust I know has enough information to prevent that.
I think that it's irresponsible to make a blanket statement like this, because there are many use-cases for multiple threads in Python. Sure, one of the obvious ones (parallel processing) doesn't work, but besides that threads can be extremely useful.
I'm also unclear on what you mean by "no such thing as thread-local data" when there is `threading.local()` that does exactly that.
Lastly, I don't think multiprocessing was created as a workaround for threading per-se. Rather it was a workaround for the global interpreter lock.
Were it my choice, any time the need for concurrency comes up at my job I'd prefer to use a statically typed, compiled langauge like C++ or Java (or, once I've familiarized myself with Rust's implementations, that language), and this kind of discussion wouldn't even come up. I like python as a rapid-prototyping language. For the kinds of numerical computations and data-laden I/O bound work I do I find it sorely lacking, and consider it an unfortunate choice for production work.
for url in list: gevent.spawn(lambda: requests.get(url)) ```
I concern this a problem already solved by gevent(or Erlang process / goroutine). Actually, I didn't see a benefit introduced by asyncio. Monkey patching seems scary but it works quite well in real projects. At least in my medium sized (50KLOC) game project.
Tasks that require bare async programming are incredibly rare. There's a small set of patterns that covers almost every application ever written, but it's not completely covered by Twister (or at least didn't use to be, there's a few years that I don't touch it).
Honestly, there's no problem with Python exposing the low level async primitives. That's good, and very pythonic. My only problem is that everybody is talking like this is the complete package and Python is now a good choice for async programming.
See lines 123-124:
https://github.com/jftuga/universe/blob/master/tcpscan.py
I have used this under Linux, OSX and Windows. It's cool to add the Thread Count field in Task Manager and then see something I wrote use so many threads! I am more of a sys admin, so this code could be better - but it seems to work very well. :-)
Use asyncio (or roll your own toolkit, or just spawn 'nmap') and you get 20000 concurrent connections.
Which is better?
Your 20000 concurrent connections with asyncio are not parallel. You can have 20000 threads if you want.
As you well know 20000 threads is not a great way to do anything. Especially in Python.
extra tip: after clicking the line, press 'y' on your keyboard and you'll get a link to the file in it's state at the current commit so future commits won't break your old hyperlinks.
He is the author of several popular projects including the web framework "Flask". This makes him a person very respected in the Python community - personally I love his taste for interface design.
I wish he could interact better with the core team, because some of his rants are not as constructive as they might be.
Can you be more specific? How could it be more constructive?
Perhaps this is me showing some cultural bias (I'm Brazilian), but in my social context it is considered rude to make this kind of criticism in public unless you tried your best to address the problem in a more restricted exchange. The more authority you have, more you are expected to come with a better proposal instead of just pointing out the mess.
This entire adventure started because I was looking for a way to get logical call contexts working in it and this shows that there is not enough internal machinery to support that at the moment.
I'm not going to fight for that however. I hope my post was rude.
That sounds like orders of magnitude more work. In case you are not interested in performing that work, it seems more constructive to me to share your thoughts rather than to say nothing.
It's not Armin's duty to interact in any way with the core team. Hell, the core team should be working to please Armin if you ask me. He's your target user, and he's disappointed with your product, it's not his fault.
I think Armin does a wonderful job providing a voice for those of us who are increasingly disenchanted with Python 3.
Gevent monkey patching isn't perfect but it works and gets you closer to how an event loop should be used with standard libs IMO, closer to Go.
This is why python is still used for it's purpose. CLI utils, bots/crawlers, tf/pandas/scikit-learn, REST-APIs on top of flask.
Then hard times came. First, many year long story (still not concluded) of transition between 2 and 3. Then all this stuff. Sure, there is such thing as progress. Stuff is invented for some purpose. But now, in 2016 and at v3.6 — Python isn't what it has been loved for. Not easy-to-start-with, nor simple. This return/yield fuck up, showed in the article is absolutely huge deal, for example, and it is not about asyncio per se. Async stuff is always complicated, it wouldn't be that bad if it was all it's about.
If some 5 years ago one would use Python just because "why not, I just need to get stuff done", now it's quite likely that after struggling with all this micro-nuisances he would go with golang/js/php/whatever instead.
This approach to async, though, is just a language feature that's becoming mainstream right now. C# has it, ES7 has it, C++ has a working paper on it etc. Python actually had the benefit of watching how things work out elsewhere before implementing it all.
For example, speaking of async - even 2.7 already has Twisted, and an ecosystem of libraries around it.
The only two ways I can see it being solved is either by making it more of a toy language (which is great if you're just writing short scripts, but it's not really what it's supposed to be about); or by having a very centralized "best practices" enforcement that basically forces libraries to conform through peer pressure, like Java - which has its own disadvantages aplenty.
If some library messes it up, and requires you to provide bytes that are semantically a string, it's a design flaw in the library.
Well, do those encode/decode calls serve a purpose? Does encoding matter?
If so, they're not cruft - they do something that needs to be done. You may dislike the fact that it's extra complexity, but any speaker of a language for which ASCII (or Latin-1) is not sufficient will rejoice knowing that you can't write code that breaks when we try to use it anymore. I remember how much of a hassle this kind of stuff was back in DOS/Win9x days, and even in early 00s; and I also remember how a hard push for Unicode as the string encoding in mainstream languages like Java and C# did a lot to rectify that.
If not, then why would you need to encode something just to decode it later, or vice versa? Why not just pass bytes/bytearray around? Again, if the library requires doing so (because they demand the data to be passed as a str specifically, even though it's never actually treated as a string), then that's a bug in said library, and you should complain to/about its authors instead.
That being said, as someone who started working with asyncio in Python 3.5 none of it feels particularly difficult to understand. Asyncio needs more work, sure, but the API so far is relatively straightforward.
The overloading of iterators/generators is a bit odd. But the same approach that was taken with JavaScript -- and it's nothing anyone working with Python 3.5 or above will be exposed to. If I recall correctly, Python 3.6 will even feature async generators. Any developers diving into this using that release won't have to ask themselves why they can't 'yield' from within an 'async' function.
It does not help that asyncio evolves in the stdlib and changes with every major Python version. It might be less of an issue if this was pip installable I suppose. Right now writing utility code for asyncio is targeting many things.
Personally I have been looking at some of the patterns taken by the team on aiohttp - I think they've done a good job and, iirc, one of their members is a core contributor to Python.
The whole experience has been mega confusing as there are so many similar yet slightly incompatible concepts.
No. It was supposed to be a language in which programmers at widely different skill levels, from beginner to expert, could be productive. Easy to pick up the basics, but also easy to use more advanced techniques when you find you need them.
[1] http://www.snarky.ca/how-the-heck-does-async-await-work-in-p...
Asyncio has a steep initial learning curve (especially in our hybrid setup) but it's well worth it. We target low resource computers like Raspberry Pi and using async over threads has speed up things a lot.
The biggest catch is that while writing code you have to think about every function that you use. Is it doing I/O, is it a coroutine or is it callback/async friendly.
Before, you needed multiple threads, or else incur high I/O costs. But then you have to manage your shared state across threads, hence locks.
Their code is now probably async single threaded code, like JS, where locks are irrelevant.
Imagine two threads running
That's four opcodes. Only one operation will run at a time, but one thread might preempt the other midway through the four operations, and `a` will have been incremented by 1, not two.In some sense, mulithreading with Python's GIL is the worst of both worlds: you can use only one CPU, but you still need to account for concurrent state mutation. (Of course, it doesn't have the function color problem of asyncio http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...)
Actually... I take that back. Asyncio is solving a different problem than the locking of shared resources.
https://github.com/home-assistant/home-assistant/blob/dev/ho...
User jerf's comment that asyncio "more than [doubles] the complexity" is absolutely correct. Watch this video of Guido talking about tulip...or struggling to talk about tulip, rather. It's clear the dude is out of his depth and my god the recent changes to the language show that the inmates are now running the asylum... Seems like Python, in it's effort to chase the latest fads, is no longer the language I would endorse to someone new to programming. Whether you think that's a meaningful litmus test or not, the staggering amount of crap that's infiltrated the language now completely flies in the face of the zen of python's statement that there should be one and preferably only one way of doing things. Fuck. I'm going to go code some lua now.
https://www.youtube.com/watch?v=1coLC-MUCJc
I agree with coleifer (and others) that Python 3 is losing "the zen". The new f-string stuff is a perfect example: a bad, retrogressive idea that never should have been approved.
I have no plans to adopt or use Python 3.
A convenience that is present in every other modern (and not-so modern) language except python? Hardly.
Interpolated strings are awesome. Only problem is that they weren't introduced in python 3.1 so you could drop the f
That said, I use Python 3 every day for almost everything without need to use every shiny new feature.
I still do quite a bit of work with python2. These days I am using Go in quite a few of my personal projects, and for work projects where I am given the choice.
I had the pleasure of working over IRC with you (at Ellington) when I worked for CMG Digital, and I found you to be one of the most knowledgeable Python developers I had met.
But...
I think you're blowing this out of proportion. I think that yes, the implementation details are far too complicated, and yes there has been some serious (IMHO) mistakes made (e.g., type hints, format strings, etc.) in Python 3, but overall the language is terrific and getting better.
My hope is that the implementation will become:
A) simpler
B) more optimized
C) out of the box functional (e.g., no need to manage event loops and other things yourself)
D) unified (e.g., the zen of python even states that there should be one way to do something)
I also strongly agree with the comment @RodericDay made herein, pertaining the new typing syntax and the addition of yet another string formatting syntax (a dangerously unexplicit one).