Well, "put your money where your mouth is" would be more apposite. I commend them for volunteering to contribute in this way. A rising tide and all that.
Isn't this just how most open source software with enterprise customers operate?
Big companies pay (money or with developer time) in order to gain support and to steer the product to add features they want. The open source produce lives on, developers get paid for their work, big co gets features they want, and everyone lives a happy life.
The editorialized title "Meta pledges Three-Year sponsorship for Python if GIL removal is accepted" seems misleading at best. The actual wording was:
> support in the form of three engineer-years [...] between the acceptance of PEP 703 and the end of 2025
It seems much more like them pledging engineer time from Meta employees to work on this project than sponsorship, not a monetary sponsorship like the title implies.
Yes, and the comment is to request help essentially:
"it would be great if Meta or another tech company could spare some engineers with established CPython internals experience to help the core dev team with this work."
That also depends on the scale of the monetary sponsorship, though. If I heard something like "Meta/Google/Whoever is the main sponsor of Python for the next 3 years", I'd assume (perhaps incorrectly) much more money per year than it takes to hire a single engineer. On the other hand, someone just saying "sure, we'll do the silver tier at your next few conferences" is worth a whole lot less than one engineer.
Regardless, the previous commenter's point stands that the title could be a lot more informative.
It sounds like Facebook is volunteering 3 person-years which is nothing resembling a 3 year sponsorship. We're talking at least an order of magnitude difference. 3 person-years sounds like one team at Facebook working on the GIL for 3 months.
It's until the end of 2025, about 2.5 years from now. The comment was by GVR, who is presumably adding up all the pledged time to estimate whether there's enough in total.
This is potentially problematic. It creates a conflict of interest: will the GIL proposal be accepted based on technical merits and potential benefits to the Python community overall, or just because the Python foundation wanted money? The problems that Meta faces in production are not necessarily the problems faced by every Python developer. In particular, Meta might value performance more than backwards-compatibility, while other Python users might feel differently. If Meta were to offer unconditional support, that would clearly be good for Python; attaching conditions to their support makes the situation trickier to navigate.
What is Meta's end game with Python? Do they genuinely just want a stable build without GIL because of some cranky in-house code piles, or is this the first step towards a bigger mission to wrangle an entire ecosystem?
Calling it a "donation" is even too generous. They use python and would save a lot of money running it (less severs) if it could run multi-threaded. Implementing it themselves and maintaining a fork may be cheaper in the short term but due to ongoing maintenance and third-party extension incompatibility it would likely cost them more in the long run. So they are effectively investing 3 engineer/years of resources to A) ensure the project gets done and B) help it get done faster.
I don't see any charity here, this is just a smart business move by them. When you run as many servers as Meta does being able to run 1/4 as many instances is huge RAM and therefore money saving. It won't take long to add up to 3 engineer/years.
I agree charity is not the right way to frame it. But if this was a pure money decision they would save time and effort forking the language and improving it as closed source.
They have the expertise to run a language fork and because of the scale they run at they would easily save more money doing so. Meta contributing to OSS is not an economic decision. It is about branding, trying to attract technically strong individuals (who tend to contribute to OSS or want to), and sticking to their open culture.
> they would save time and effort forking the language and improving it as closed source.
Actually, experience has shown the opposite to be true. Time and time again even large organisations find their fork falling behind the community version and the pain of merging new versions from upstream grows and grows
GIL has been a major pain point when training machine learning models, where every library is Python first. The data preprocessing phase must happen quickly enough and in parallel with respect to the GPU operations. The simplest option is doing data processing in several threads other than the one driving GPU operations, but that is not possible with GIL. TensorFlow and PyTorch each has their workarounds for this problem, but many difficulties remain. Removing the GIL will be greatly beneficial to this domain at least.
Because the whole point of the thread is "It creates a conflict of interest because money" which it doesn't. Time costs money, sure, but time is not equal to money. There are many differences. Otherwise where is my money from HN.
Woah, are you kidding? I would have died to have people dedicated to OSS I worked on! Time is money. We get paid for our _time_ at the core of what we do as a profession. This should be plainly obvious.
The alternative is Meta puts in no money to salaries and benefits for up to three individual's time on this project, or Meta puts in no money at all.
Meta is paying, one way or another, for this project.
> Meta is paying, one way or another, for this project.
Did I deny this?
To put it in extreme clear example, just said time for someone to punch you != time for someone to work for you, while money is same all across. I hope you don't disagree with it.
And it's 3 years of engineer time to work on this project, so I don't get the bribery insinuation. Engineer time to work on a specific project is only valuable if you consider the specific project valuable.
You could maybe make the argument that they're trying to swing priorities if this is considered valuable but less valuable than other things, but given the PSF doesn't actually employ anyone who it can direct to work on specific projects anyway, how is this any different from the already existing prioritisation process of "things that somebody is interested enough to go through the PEP process and then work on"?
TBH having CPython internal experiance is quite a qualification. They would probably contribute at least some of thier time in the no PEP 703 case so the delta is even leass.
It's three engineer years until 2025, and it's just a person mentioning on the discuss board. It sounds a bit weird to throw something like this in a public forum.
Why weird? The request was made there. The discussion on the PEP is there. And the person answering is a core Python dev and part of the Python team at meta/insta.
Their team wants it, and is offering hours to help get it done. Is this not what everyone wants from businesses using OSS?
Python is closer to C than we all like to imagine, in more than one way. Lots of access to low-level libraries. Long history of writing modules in C. And little in actual memory safety protection more than relying on programmers to please not write bugs in their C or ctypes code.
Philosophically, Python is close to C in the sense that low-abstraction interfaces to system functionality (file descriptors, memory) exist. Numpy allows access to uninitalized memory and you can manipulate how numpy uses memory (mmap, shared memory, sharing memory between objects). Ctypes allows access to malloc and dereferencing dangling pointers. This is not exactly normal Python, but it's inside the realm of the possible in Python.
All of this makes me worried that the challenge of "free threading" is quite large. I wonder if the subinterpreters solution offers a safer route. At least it's not pulling off the GIL band-aid all in one go.
On other hand, allowing access to free threading would be a straight-line continuation of this culture: just make that feature available. ("we're writing system code in a high level language.")
They committed three person-years, not three years. This project would need dozens of person-years over a three year period.
Also a comment on Github is not a binding support contract. Meta executives could deprioritize this project at any time... hell the person making the commitment could already be laid off, we have no clue. As someone who worked in big tech for a long time, trust me - it needs to be in writing with an exec signature.
It’s an interesting debate ( money aside ). In the past 15 years, at least to my eyes, one of the most important changes in soft eng culture has been the now systematic search for correctness. Systematic, organized, large scale Testing is one side of the coin, but the real value , and significantly more important change to me is guarantees/proofs : static analysis, stronger type systems, etc. So type annotations in Python, typescript, rust instead of c++, etc. Tools as well : golang race detector, infer, coverity, pvs studio, code peer (Ada)/spark, compcert, etc.
In this particular case, I understand that removing the Gil would create potential new risks ( but provide better performance) and although tooling is mentioned, it’s not officially part of the plan, so it feels like a regression of some sorts.
It might be worth it though because somehow python had ended up being the lingua Franca in a many applications where performance matters a great deal (ML, scientific applications, etc). I don’t think the performance benefits for something deployed at such a large scale should be understated.
Although you are almost certainly correct, I’m curious if there is a case where faster doesn’t mean more power efficient I.e. more concurrency but more power overall
Concurrent execution is ~always less power efficient than serial execution. I have no idea what the parent comments are talking about. For almost any work X with time to run on a single core T it will run on two cores in time larger than T/2, and on three cores in time larger than T/3, ... This is due to synchronization overhead (e.g. networking, locks, delays) and also often due to shared resources getting saturated (e.g. VRAM on a GPU).
That is unless in the serial execution you still have to pay for some resources that are unutilized.
However, if you're running a computer with a 16-core CPU, power usage doesn't scale linearly with cores. There's a lot of overhead, especially if you're talking about a laptop/desktop with display, HID, etc.
I don't think that has really been a change in the past 15 years. Well, maybe there was just a lull in people caring about correctness and robustness... But it's not like before 15 years ago everyone was using dynamically typed languages and static analysers didn't exist.
I guess you could say the world went through a dynamically typed loosy goosy phase and then realised that wasn't so great.
I think one thing that actually has changed is that SMT solvers have massively advanced in the last 20 years to the point that formal verification is practical. Ish anyway. It still seems to require a PhD to do software formal verification. Hardware formal verification is easy though.
The FAANG etc failed me for anything open source sponsorship related.
Their business depends on it, they made a fortune out of it, yet they donate little or nothing to it, or worse, they forked it and was reluctant to upstream anything.
pick Amazon randomly, it went from exploiting to rip-off in my opinion, thanks to its boss , who sails on a luxury yacht and asks all his employees to be frugal. they all look similar to me as far as supporting OSS goes.
React alone is one of the biggest feast Meta brought to open source community, and shaped the web from monolithic pages (back then PHP and jQuery is the majority) towards modular components design. Later they even introduced functional programming paradigm in the mix.
Next.js exists too which mashed up PHP/Ruby style with React.
In terms of companies that rip off open source and do not give back meta is certainly one of the more active "give back" companies. React/Native, docusaurus, buck, pytorch, rocksdb...
yes some big companies did open source projects and Meta is a good role model, for many those OSS projects, they work for the companies themselves first then the community, though they are indeed still 1 million times better than closed and proprietary code.
I'm mainly referring to long standing fundamental projects that made the rest projects possible, not those projects that are created and managed by the big companies themselves.
Traditionally speaking, it's because they're the most qualified to do it. Every FAANG company takes from Open Source in order to succeed, so contributing back to a FOSS codebase is the ultimate act of corporate goodwill. You are burning real-life money and engineering hours to fix a problem for everyone and ensure that it stays protected under the same Open license that let your company succeed.
There is no legal obligation per-se. Open Source is a charity, but you can understand a lot about a company's motives if you compare their FOSS usage to their FOSS contributions.
I think this misses the point. FAANGs don’t improve OSS code out of goodwill, they do it because it benefits them. Upstreaming those improvements has very little marginal cost, and it bolsters an environment where other companies do the same. Rising tide and all that.
It’s not a charity if it becomes or serves as part of a standard. Sort of how semis and the various suppliers/buyers of that industry wind up agreeing on standards despite being competitors in some cases. Definitely things like the Linux kernel make sense to give back to - you’re ensuring the kernel lines up with your own corporate investments in a sense. A company unilaterally needing to fork and maintain their own Linux compatible kernel would need to sink a lot of time and money to come up with something as developed as the modern kernel. And it would become more expensive, unwieldy, and akin to technical debt over time.
It’s clearly not perfect calculus since so many defect from OSS, but there’s also clearly some business sense there.
> It’s not a charity if it becomes or serves as part of a standard.
Then it just transitions to a charity that everyone relies on. It's been embarrassingly difficult for people to admit this vis-a-vis Linux and the like, but it's obvious in a situation like programming languages. We rely on the enduring charity of it's authors, license-holders and distributors to write and distribute software.
"Giving back" is not an inherently cynical process. Meta has no obligation to publish these patches in the first place, much less work as a part of the community to solve a common issue. But they do, and it's just kinda part of their culture at this point. Yes, they do still gain from it; but so do their direct competitors. It's difficult to characterize as a purely business-motivated decision, because there are much greedier avenues to pursue if that's their MO.
Why would they? Nobody except a few home users needs RAID 5/6. There's only a small handful of talented filesystem developers (never mind the filesystem, it's equally applicable to all of them), and there's always something to do which will benefit all of us. Especially for something so feature-packed and complicated.
"Giving back" might be an understatement for Meta's open source contributions. They are the bedrock of so much OSS in the last decade, it's everyone _else_ that is "giving back."
It is all about knowing individual engineers inside these companies. Nearly any engineer in FAANG can, with a little effort, donate their own work time to an opensource project in the form of submitting patches.
The project can help in various ways. Make sure licensing stuff is in place so that whatever IP review process the company has goes smoothly. Make sure PR's are reviewed fast. Delegate responsibility ('hi, would you like to be maintainer of this feature?'). Make sure contributions are publicly recognised (ie. Public page of maintainers/biggest contributors).
Remember that most engineers inside big companies are trying to generate good content for their performance review. "Wrote and open source some code" is usually one of the checkboxes.
> Nearly any engineer in FAANG can, with a little effort, donate their own work time to an opensource project in the form of submitting patches.
I know at least one of those letters that does not want you doing that without jumping through hoops and getting prior approvals. I will let you guess which one that is.
Thats the "with a little effort" bit. But as long as you have a business case (eg. "we are adding this feature for our use, but it will lower our costs if it is upstreamed rather than maintained as a private patch"), it will usually be greenlighted.
No, Meta’s contract has explicit carve-outs for Open-Source work done on personal time that are more generous than Canonical’s (I had to choose between the two 3 months ago).
Meta has its own production fork of CPython, Cinder:
This isn't legal/enforceable in California. Your own work on your own time on your own equipment is your own property. It's messier if you're building a direct competitor or using proprietary knowledge.
It's easy to blame the boss, but I don't think its the companies responsibility to give back to open source just because they use it. If you think open source is helping you, you could contribute it in your time without expectation to get paid for it. Most open source projects work like this. And if some company needs some extra feature, they should pay market rate for it.
Assuming the company doesn't have clause to not allow contributing to open source. These companies should be publicly shamed if they use any open source code.
From the many people who asks for things, only a tinny percentage are actually willing to pay for it. Take Microsoft, I had one of their employee asking me to support their azure stuff: https://github.com/mickael-kerjean/filestash/issues/180. When I found out the dude was actually employed by Microsoft, I somehow though he could give a hand, but no he was just there to make some feature request ...
I had no problems with this by the way, just saying maybe he can contribute 0.001% of his profit to those OSS projects that he depends on, it will help him to build more yachts too.
Between pytorch (Meta) and TensorFlow (Alphabet), I'd be surprised if there's much if any ML work today that doesn't rely on OSS projects from FAANG companies.
Chromium underpins nearly every major browser that isn't Safari or Firefox. And even more narrowly, V8 powers Node (and Deno).
More Google contributions: Angular, Kubernetes, gRPC, Golang.
Microsoft: C#, TypeScript, Language Server Protocol, VSCode.
Meta: React, zstd, Apache Cassandra, LLAMA.
This isn't even getting into open source patches (Google has consistently been one of the top 10 Linux kernel contributors for the last decade), systems papers that have inspired research and other systems (BigTable, Dynamo, MapReduce, Colossus, Spanner), standards work (HTTP/2 and HTTP/3 were both adapted from Google technology), security vulnerability work (Project Zero).
(The above list is definitely Google-skewed because that's what I know.)
Chrome, though, is a kind of mixed bag. It already has too many features, most likely the reason it underpins so many other browsers is that nobody can keep up with the rate at which Google shovels shit into there.
They should slow down. It would be nice if they could just spend a couple years working purely on stability/performance/security, but alternatively they could just stop, it would be better for the ecosystem.
A lot of these (but not all) strike me a flashy things for which you get promoted. Many people don't need any of these.
What FAANG does not do is support critical yet underappreciated Unix infrastructure. By this I mean directly donating to developers without trying to take over the projects.
The quality of FAANG code, especially in the fault-tolerant machine learning space, is questionable as well. I'm not sure why so many commenters (in other comments!) attribute near magical qualities to any output of FAANG.
Even stuff like UNIX, C, C++, to extend your example.
People apparently forget the amount of money that was and is injected into them.
Someone has to put the money on the table to attend ISO meetings, buy the standards, implement the features into existing C and C++ compilers, for example.
Talking about fundamental OSS projects that is not for-profit at all.
Many of Google's OSS projects are there to help its profits, if not, Google could stop or kill it at will which is fine and is 1 millions times better than closed code, still not the same as projects like python etc.
yeah i feel like there are tons of reasons to criticize these big companies, but open source, theyre pretty good for the most part. not sure what else you could expect from them.
there is graphql, rocksdb and folly. they also open sourced their js engine for react native too.
maybe a good criticism is we depend to much on these companies free stuff lol
This is disappointing but certainly not surprising and to some part we as an open source community have to blame our selves.
I know it is not a popular opinion but the AGPL is there for a reason. If you release your code under a permissive license you should not be surprised that others benefit from it - sometimes massively. That is how it is supposed to be. You can still lament that it is unjust and I feel you, because in a sense it is. If it really bothers you, the solution is easy. Next time choose the AGPL.
I've released open-source projects under an Apache license because I want others to benefit from my work. I knowingly didn't choose GPL-flavored licenses because they limit people's ability to do this.
It's stupid sour grapes comments like this that make me hate HN commenters.
Meta and Google have both contributed an enormous amount to open source (creating PyTorch, TensorFlow, Chromium, contributing to clang/LLVM, Linux kernel, HTTP standards), but you're mad because they didn't sponsor your work and a bunch of people mindlessly upvoted it.
Whatever happens, I'm hoping we won't see another 2to3-like migration. I moved from Python a long time ago due to its multi-threaded limitations (among other things) to Java and JVM, and I really appreciate the stability the JVM team tries to maintain. Things are not perfect, but I'd take stability/compatibility over perfection any day.
Could not agree more. The purposeful unnecessary breaking stuff was maddening- and then being lectured at like you were an idiot more so. No more u”” for you!
There is zero chance of that happening. Guido van Rossum (and other key figures in the Python community) have spoken very candidly for years about the mistakes they made in the 2-to-3 migration and their decisions in the year since have demonstrated their commitment to not repeating these mistakes.
I was curious about these comments and did a small web search. I found a video interview from May 21, 2021 [1] and have pasted some excerpted quotations from it from Guido van Rossum below for others who are curious.
“Python 4, at this point whenever it’s mentioned in the core development team, it is very much as a joke… We’ve learned our lesson from Python 3 vs 2, and so it’s almost taboo to talk about a Python 4 in a serious sense.”
[...]
“I normally talk about that as a mistake, because Python was more successful than the core developers realised and so we should have been much more aware and supportive of transitioning from Python 2 to Python 3”
[...]
“I’m not thrilled about the idea of Python 4 and nobody in the core dev team really is – so probably there never will be a 4.0 and we’ll just keep numbering until 3.33, at least”
[...]
“We now have a strict annual release schedule, so after 3.10 will be 3.11 and after that will be 3.12, and so forth. We can go up to 3.99 before we have to add another digit. Adding another digit is not completely trivial, but still much better than going from 3 to 4."
At least take a look at PEP-703[1]. It will answer most of your questions. Significantly, even if this proposal is accepted, the GIL would remain enabled by default.
I think at this point it is better Meta to ditch python and invest in a new programming language that mimics exactly wording for wording of python but without GIL. Google did it with their Carbon for C++, Go for C/D/C++. Microsoft even wrestle with Java for .NET and at one point their J++ 2 decades ago. Meta shpuld create their own. Maybe dump their resources into Modular. Python at this point is basically a C++ with a hotch potch of features and codebase. Start from clean like Mozilla Rust would help entire programming a lot than invest in a niche subset feature of Python which they dont have any control at all (I am speaking from corporate point of view). Python is like Perl, it will continue to have its place even decades later to be generic or niche for AI. But the rest of us may be moving on to something more specialized and cleaner like Rust, Ruby, Modular, etc.
Agreed, they should port their AI stacks to a functional language. It looks like these efforts are designed to prop up Python for the scientific stack.
Most of this work has been on Javascript, which is single-threaded and less mutable than Python, but PyPy has also made some significant contributions. PyPy has a GIL.
There is also work on Ruby. Ruby is even more mutable than Python, but also has a GIL. Work on Ruby mirrors that on Javascript.
I think the problem there is even a company like Meta wants to rest on the shoulders of the giant Python community. It makes sense for Meta to try to get parts of Cinder into Python.
why not just use an existing language? ATen (pytorch’s autodiff engine) is written in C++ anyway, so why not use that?
i really don’t understand the trend with launching a new language every time it seems convenient; it may have the advantage of being made at {company}, but this is also its great downside: it exists, is maintained, and used only with relation to the project/company that prompted its creation
for example, swift is only used for swiftui and some other apple specific stuff, and no sane person is going to write an unrelated serious project in it
Perl never had anywhere near the wealth of libraries and community support Python enjoys today. The language is used nearly everywhere, from utility-scripts to build piplines, desktop applications, webservice backends, all the way up to scientific computing and AI.
Python shines because of it's ecosystem and community support, which I'd guess (without having any numbers to back this up) may very well be unmatched among programming languages today. Maybe Java and C/C++ come somewhat close, but I'd doubt even these behemoths would stand a chance in a 1:1 comparison for sheer size and versatility of what for Python is just a quick `pip install` away.
Any language that want's to replace Python, even internally at a company, will either have to deal with the fallout of leaving all that behind, OR be compatible with Python to an extend that it doesn't matter. And that's a really tall order.
Almost every single data structure access in python, even reading a dictionary item, at some level depends on the GIL for true correctness right now. It is very deeply embedded in cpython.
I think it’s the other way around? I haven’t run into any ML code that could be multithreaded that wasn’t written in C++, but have often run into server tasks that could use a polling thread, etc.
All the ML code is written in lower level languages and that’s very unlikely to change, GIL or no.
Yeah, you're right - even though CUDA is async, doing any preprocessing (in Python) can be harder if you don't have shared memory (the start-up latency hit of multiprocessing is not a problem in this context). I've only ever encountered "embarrassingly parallel" data-feeding problems, where the memory overhead of multiprocessing was small, but I could see other situations. Comment retracted.
The GIL problem won't be solved by throwing engineer hours at it (not even Meta engineers). Fundamentally every single piece of python code ever written will have to stop and now worry about potential race conditions with innocent things like accessing a dictionary item or incrementing a value. It's a massive education and legacy code problem at least on the same scale as the python 2 to 3 migration.
I honestly don't think the community is ready for this change and don't expect to ever see stock cpython drop the GIL--perhaps there will be a flag to selectively operate a python process without the GIL (and leave the onous on you to completely test and validate your code works with potentially new and spooky behaviour).
I dont get this. Just because you don't have the GIL doesnt mean that your previously single threaded code is now multithreaded and stepping on itself.
From previous discussions it's my understanding the c integration is going to be the cause for the issues.
From a python perspective it wouldn't necessarily be a big change, but everything can branch out to c, and there you're going to get in trouble with shared memory.
The most significant impact of this change is that python threads can run at the same time. Before, calls to C API were essentially cooperative multithreading yield points - even if you have 100 python threads, they can't run concurrently under the GIL. 99 are blocked waiting on the GIL.
C extensions have always been able to use real threading. But now there is no GIL to synchronize with the python interpreter on ingress/egress from the extension.
No, but it means that your previous multithreaded code is no longer automatically prevented from stepping all over itself by having multiple threads accessing the same data:
GIL removal doesn't mean "make all of this lockless" it means "replace the GIL with fine-grained locking". So those problems are still solved for Python code. The three issues are the amount of work it takes to do right, the performance cost for single-threaded code, and the CAPI.
finally. you don't even have to read anything to work this out -- if things like dictionary access were no longer atomic that would imply that threaded code without locks could crash the interpreter, which isn't going to happen.
I'm simultaneously scared and enlightened seeing all these comments acting as if the GIL is some magic "makes your code thread/concurrency safe" pancea. I always saw it as a shim/hack to make cpython specifically easier to implement, not something that inherently makes code more thread safe or atomic. It's just more work to do things "the right way" across application boundaries, but from my understanding this PEP is Meta commiting to do that work.
Removing the lock creates problems in existing code in practice. This is an ecosystem that has less focus on standards and more on "CPython is the reference implementation".
What non-transparent GIL specific behavior are developers relying on exactly?
When I say GIL specific behavior, I mean "python code that specifically requires a GLOBAL interpreter lock to function properly"
Not something that simply requires atomic access or any of the garuntees that the GIL has this far provided, but like, specifically code that requires GIL like behavior above any CPython implementation details that could be implemented with more fine grained concurrency assurances?
I've seen some really cursed python in my days, like checking against `locals()` to see if a variable was defined ala JavaScript's 'foo in window' syntax (but I suppose more portable), but I can't recall anything specifically caring about a global interpreter lock (instead of what the GIL has semantically provided, which is much different)
> What non-transparent GIL specific behavior are developers relying on exactly?
They are relying on behavior of a single environment. We similarly see a lot of issues moving threaded code to a new operating system in C/C++, because the underlying implementation details (cooperative threading, m:n hybrid threading, cpu-bound scheduling, I/O and signaling behavior) will hide bugs which came from mistakes and assumptions.
> GIL removal doesn't mean "make all of this lockless"
Literally speaking, that's exactly what "removal" means. As far as I can tell, GP was wondering why there's so much discussion about replacement, since simply removing the GIL wouldn't break single-threaded code.
Then just mark the extensions that is compatible with GIL. And also you will have a switch that disables GIL, controllable with a environmental variable or launch option.
Yeah, I'm not a full-time developer but even for my bits of scripting I'd be wary of another big change in Python.
Personally I'd much rather see this effort go towards a new language which "feels" like Python but adopts more of the development experience of Go and Rust. From my tinkering it seems like Nim might already be that language, in which case what is needed is investment in its package ecosystem.
The problem with Python 2 to python 3 is that python 3 was essentially a new python 2 esque language instead of just being a major version bump.
If python 4 was no GIL python, then both Python and C would remain unchanged as a language.
It will be optional. You don't have to worry about it, set the option to ON and for those who can worry about it, they will have the option to set it to OFF.
The problem is, code that uses on a lot of 3rd party libraries that throws the ON-switch for nogil, will suddenly depend on all these libraries maintainers having worried about this.
We can set it up such that if a module imports modules that don't support nogil, nogil will be automatically disabled for them too.
So, library designers will be under pressure to update their libraries to enable support. We could also have code tools that detect patterns that aren't GIL safe and throw out loud warnings
Why should library authors be put under pressure because someone else chose the wrong tool for the job, and is now trying to push that externality on to the community?
Python is a single-threaded language. That’s part of its DNA. The community has already been through one traumatic transition in recent history and the appetite for another one is low.
Library authors should not update their libraries to support multithreading, rather the people who want that should be forced to rewrite their code in a language that is more suitable for the problem they want to solve.
I disagree that single threadedness is in its DNA. It's an implementation detail of CPython. There are other implementations which don't have a GIL even today.
Would removing the GIL be a big change for CPython? Yes. But IMO it's worth it
> Python is a single-threaded language. That’s part of its DNA.
Boooo. Maybe if all you do is ops scripts, but for those of us in data science this couldn't be further from the truth imo.
I'm not saying the syntax is easy for asyncio, or anywhere near as nice as golang or even kotlin is for concurrency, but it's definitely workable in a concurrent environment.
> Python is a single-threaded language. That’s part of its DNA.
If the changes proposed in this PEP go through, that will no longer be the case. So library authors pretty much will have to either update, or see their modules wither the same way they would if they weren't updated as new Python 3.x versions come along.
> rather the people who want that should be forced to rewrite their code in a language that is more suitable for the problem they want to solve
The vast majority of the Python developer community WANT Python to support true multithreading and being able to solve these problems. We expend inordinate amounts of time and skill trying to make Python work around the GIL, eg. by utilizing multiprocessing.
We want Python to stay relevant long-term. In an age of abundant multi-core platforms, and workloads that can utilize them, the GIL is a major obstacle to that desire.
This is wrong, that ship sailed 10+ years ago. Python is used for almost everything, better get used to it.
This kind of garbage logic lead to the horrible Ruby/Python/Javascript with C extensions split that makes their ecosystems very brittle, versus Java/C# where it is expected that things are fast enough without C and package management is much easier.
It's true that python is used for "almost everything", but it's only true because it plays nicely with C.
I understand the desire/demand for general purpose tools. The thing is, there are always tradeoffs. Acknowledging the tradeoffs and designing more specialized tools that work well together isn't necessarily garbage logic.
Getting rid of the GIL would warrant the release of Python 4.0 for me, except the Python project shouldn't be supporting two different branches for as long as they supported 2.7.
I imagine there would need to be some kind of annotation to enable the GIL for a method and all of the code it calls, including libraries, so performant Python can take advantage of the lack of a GIL but old code doesn't break. Then all you need to do to maintain compatibility is to annotate your main() and your code should remain compatible for a while.
After all, the referenced PEP explicitly calls for making the GIL optional, not for removing it completely.
>Python project shouldn't be supporting two different branches for as long as they supported 2.7.
That wasn't the problem. The problem was not giving people the ability to make their code python 3 compatible while they were still stuck with python 2. The python 3 interpreter should have had a python 2 mode that gives you warnings.
That is a misconception. The GIL protects the internal state of Python. It don´t make all Python multi-thread code "safe".
The PEP 703 still preserve many actual characteristics such as access and writing to dictionaries.
Correct. Python multi-threaded code is not magically thread-safe. What it did do is make C functions atomic, like dictionary and list methods, since they're implemented by C functions. The reason is the GIL is released and re-acquired while executing Python code (every 100 ops if memory serves), but isn't released by most C functions, making them called with the global lock held, which makes them atomic. You can still have a thread switch between any two Python op codes. e.g. foo += 1 may or may not be atomic depending on how many op codes it compiles to (IIRC, in CPython, it is not atomic.)
One of the reasons I love using gevent (https://www.gevent.org) is that it's way of introducing concurrency that does preserve these atomicity guarantees! Broadly speaking, the only time your code will be interrupted by something else in the same process is if some function call yields control back to the event loop - so if your block is entirely made up of code you control that doesn't do any I/O, you can be sure it will run atomically. And you get it for free without needing to rewrite synchronous code into asyncio.
This does make me wonder, though, if gevent will survive PEP 703's massive changes to internal Python assumptions. That said, gevent does work on pypy, so there's some history with it being flexible enough to port. Hopefully it won't be left behind, otherwise codebases using gevent will see this as a Python 2 -> 3 level apocalypse!
Ah - it's more that you wouldn't typically need to run multiple threads in the same process to handle concurrent requests. For instance, gunicorn used with gevent workers will typically fork processes to ensure all cores are used, but wouldn't require multiple threads per process - gevent would handle concurrency on a single OS thread within each process.
As soon as you leave python, that C extension can give up the GIL and your other python threads can start running.
The GIL makes python code have cooperative threading. It does not protect from e.g. your thread's view of state mutating when you make a database call.
I also believe it is best practice not to mutate data without holding the GIL in extension code, not a requirement - but I have mucked with a lot of different extension API so I might be confused.
No, I wouldn’t call it cooperative threading, it’s still preemptive in that any Python thread can be switched with another at any instruction. That’s the same behavior as the operating system with the same potential for race conditions (except python instructions are higher level than machine code instructions.)
While C extensions can release the GIL, that only makes sense if they do enough work that a Python thread could get some things done in the meanwhile, and it wouldn’t be surprising to the caller. Obviously the C thread can’t interact with the Python world after the GIL has been released.
It is solvable by making the hard decision to move to Python 4 with no backward compatibility. The two core issues imo in Python are the GIL and the environment hell and both simply can’t be solved while still keeping the 3 moniker. We’re in a field of constant workarounds and duct tape because we try pleasing too much
Python tried that (version 2 to 3) and both the community and dev team were traumatized by the effects enough they've publicly said it'll never happen again.
That means they didn't learn from it at all. The problem with Python 2 to python 3 is that it lost backwards compatibility because of very silly reasons like turning the print statement into a function. The vast majority of the problems could have been avoided by not making pointless changes with dubious benefits.
I seriously doubt anyone had problems fixing print as a statement. 2to3 fixed it...
I'll admit that, yes, changing string to bytes and unicode to string was a bit annoying, but the change itself wasn't fundamentally 'of dubious benefit', it did have benefits, and related to this, the only major issue was that you couldn't, for a long time, have code that worked in both where it came to literals. The biggest problem here was the implicit conversion from 2, that I agree needed to go.
Most of the other things can be trivially fixed automatically, or at least detected automatically, but without type hinting, it wasn't really easy to fix the automatic conversion.
There were other changes that were a bit tricky, but the majority of issues stemmed from the str/bytes change.
> perhaps there will be a flag to selectively operate a python process without the GIL (and leave the onous on you to completely test and validate your code works with potentially new and spooky behaviour).
Worked for ruby. The original interpreter MRI has a GIL too. Rubinius and JRuby added multi-threading with limited amounts of pain and people fixed libraries over the years. Sometimes just sprinkling lock blocks around a particular FFI calls or only doing them from a dedicated thread will do the job.
The GIL only protected you during any of those operations, so you can still switch threads waiting between LOAD_FAST and STORE_FAST and have a race.
There are a lot of things to be worried about with the GIL conversion of new race conditions that could happen, but there's already too much misinformation out there about the GIL, let's not spread this one even further.
Having worked on moving a proprietary language from its own green threaded VM to the JVM, and working on TruffleRuby I’m saddened to see this FUD still being trotted out. The GIL and similar mechanisms do not make your code thread safe. It _might_ save you from a small set of concurrency bugs, but they are fewer than you might think, and mostly it will just make intermittent existing issues that little bit more obvious when you move to real threads. Occasionally we would need to fix something in a core library or add a mutex, but those bugs could often be seen in a stress test with green threads or Ruby’s GVL.
My guess is the GIL or smaller mutexes will be needed for C extensions and a few other areas, but it’s also likely that could be moved to an opt in mechanism over time.
> Fundamentally every single piece of python code ever written will have to stop and now worry about potential race conditions with innocent things like accessing a dictionary item or incrementing a value
Not really, just make those operations atomic or have automatic locking
The GIL is Python’s global interpreter lock which bottlenecks performance in multithreaded applications by locking variables. They’re working on removing it but it introduces race conditions if programmers aren’t aware of the change.
Do you have JavaScript disabled? It is a deep link into a very, very long forum thread. Instead of pagination, comments outside of a certain window are loaded as you scroll.
Should take you to the first post, which is an overview of the proposal problem and potential solutions.
Mobile Chrome renders the page just fine. No idea about what other WebView you're using, but JS definitely needs to be enabled to load additional content.
In short, GIL is the global interpreter lock, something that is innocuous in single threaded software but a bane for performance in a true multithreading environment. I'd recommend reading up on it elsewhere if you're not familiar with it, as the post elides a lot of detail (the audience is the forum is, after all, people already familiar with python internals).
"nogil", therefore, is a shorthand for CPython running without the global interpreter lock. The crux of the proposal is a discussion on how to achieve it without making large sacrifices to reliability or performance in single threaded scenarios.
JavaScript enabled. Like I said, first site that ever behaves like this; three quarters of the Internet would be broken if I didn't have JS. (And Firefox and archive.is also run JS afaik)
> No idea about what other WebView you're using,
Android default, so also chrome-like, just not the proprietary Google Chrome browser
> "nogil", therefore, is a shorthand for CPython running without the global interpreter lock.
Ooh, I should have realised what nogil means. If this had been in the headline with capitalisation like noGIL, I wouldn't even have needed to click to understand what it is Facebook wants to work on. Thanks!
20+ years later, and billions of lines or python code later, the GIL discussion is still there. Kind of an amazing mess of a problem… mainly shows the priority of python being a ducktape type solution from the start.
Latest decade in software engineering indicate that multi-processing is much preferable over multi-threading. It's the architecture Edge, Firefox, and Chrome uses. So the question is what use cases multi-threading address that are not already covered by multi-processing? The memory overhead that multiple processes cause is relatively insignificant on recent machines with 32GB+ of ram.
Tons of use cases! There are lots of concurrent algorithms that use locking and shared memory instead of message-passing. Matrix multiplication or tree search come to mind.
Parallel blocked n^3 matrix multiplication doesn't use any synchronization primitives (short of join). The processes read from the same input matrices, but write to non-overlapping regions of memory. This is easily accomplished using posix shared memory. Indeed, this is how it is done when the matrices to be multiplied are too large to be handled by single worker nodes.
I don't think anyone is unilaterally going to move to multiprocess as a programming paradigm. Web browsers have their own reasons for using Multi-process, including security from memory isolation, resilience against crashes in the client side code, killing individual tabs, and so on.
I'd wager the vast majority of programs being written that need concurrency or parallelism still use multi threading or SIMD.
Browsers use multiple processes for security and reliability, not as an alternative to multi-threading. They extensively use multi-threading for performance (as does a lot of other modern software).
If anything, we've seen that the memory overhead of threads is unacceptably high, that's why everything these days uses thread pools / green threads / async. Multiprocessing is a niche - way too much memory usage, way too high communication cost.
> If PEP 703 is accepted, Meta can commit to support […] between the acceptance of PEP 703 and the end of 2025
Meanwhile, the Steering Council is dragging their feet, even though so many people in the community have chimed in their support for nogil being a thing (in the linked thread and two others).
This is a decision with huge ramifications and the PEP was submitted only 5 months ago. Caution is welcomed here. Specially with PEP 684 around the corner.
> The PEP was posted five months ago, and it has been 20 months since an end-to-end working implementation (that works with a large number of extensions) was discussed on python-dev.
I would expect the SC to be active participants in the community, including the python-dev mailing list, so they should have known what this change means by that point.
I think comparing it to something like nodejs the way they work around the issue is that almost all their code is written in JavaScript itself. Python has a different model where a very large amount of the code is written in c and python itself is just a glue.
A huge amount of NodeJS is written in native code. JS is single threaded. Async work doesn’t work without native code. JS doesn’t need a GIL because it only has one thread.
You use smaller scale locks as and when you need them, and you spend a lot of time to carefully implement core threading constructs and similar to make sure they are correct.
Most of the time in an interpreter you’re iterating through the byte code for a function, and mutating variables. It doesn’t matter if two threads are executing that same byte code, it only matters if they are mutating the same object.
That’s not to say there aren’t some operations that need coordination across all threads. For that you implement safe points so you signal that something has to be done, wait for all threads to reach a safe point, do the work, and then release the threads again. You can often be even smarter about this and avoid stopping all threads at once but instead stagger things.
Allocation and GC you can make more thread local to avoid stopping the world too often.
The GIL is a conscious trade-off between contention/performance against a simpler code.
Without the GIL, you'd still need to use different locking/synchronization mechanisms to ensure thread safety (if required), but they'd be more nuanced and spread over the entire codebase (complicating the backend).
Ruby [1], JavaScript (when running in NodeJS) and PHP are single-threaded. Perl is single threaded as well but it features something like to what is being proposed in PEP 554 (https://peps.python.org/pep-0554/).
I know Clojure is often described as a dynamic language and that it has some concurrency primitives, but my knowledge ends here.
[1] Just like Python, Ruby does have implementations that allow multi-threading.
Ruby is not single threaded even in its default implementation, but it does have a global value lock that is equivalent to the GIL. Not only does it have threads, it also has fibers which allow for more cooperative scheduling.
Bytecode vs JIT doesn't really make a difference. But taking to question to managed languages (so Python, Erlang, JS, Ruby, .NET, JVM etc) - there isn't really a GIL problem to start with in language design, there's just design choices wrt what, if anything, to do about parallelism/concurrency. For example Erlang and recent browser Javascript skip shared memory parallelism and use message passing (JS web workers and Erlang processes). Python's approach is motivated by it's simple refcount-based garbage collection, other GCs have it easier.
I can't even imagine how many engineering years it would require to fix just the most used libraries that expect the gil to exist. I assume 3 years wouldn't even cover the actually interpreter, throw in the library piece and how is this useful?
I can’t imagine you’ve read the proposal with a comment like this. The interpreter is already patched (twice in the proposal, for two different versions of Python), and Sam Gross has personally already patched many commonly used Python libraries. Here’s numpy patched, a mess of C and Fortran written for high performance code: https://github.com/colesbury/numpy/commits/v1.24.0-nogil
There's a great talk by David Beazley (I don't remember which one) that he basically explains that removing the GIL isn't difficult at all. After all, it's just one lock.
The issue is all the libraries and packages that were built around it. It's now been YEARS of building on top of the GIL.
So anyways, PEP 703 is a a nice effort, but I doubt we (everyday mortals) can enjoy it. Meta, and big companies with a specialized team might be able to exploit it by making sure their entire stack is GIL-free.
> So anyways, PEP 703 is a a nice effort, but I doubt we (everyday mortals) can enjoy it.
I'd suggest we can be more optimistic. There is a lot of python, regular users write daily, where we delegate orchestration to established libraries - asyncio/web-frameworks/pytorch. The GIL limits how much they can parallelize your code, and its removal will help with that.
2016. A lot has changed since then and AFAIK numpy and other most popular libs with bindings in C have an experimental branch that is already nogil compatible.
> Meta, and big companies with a specialized team might be able to exploit it by making sure their entire stack is GIL-free.
Not really. Most of the popular libraries will have nogil versions ready for sure (numpy, pandas, pytorch, etc). Server applications will probably benefit from being able to serve multiple requests in a real multithreaded environment without many changes (in fact, I bet a lot of HTTP libraries will see simplifications due to not needing to deal with processes for real parallelism). Even packages without changes will probably have at least a small degree of thread safety (for example, two instances of the same object should be independent and could be used in different threads) that allows users to still leverage parallelism.
657 comments
[ 2.8 ms ] story [ 519 ms ] threadBig companies pay (money or with developer time) in order to gain support and to steer the product to add features they want. The open source produce lives on, developers get paid for their work, big co gets features they want, and everyone lives a happy life.
> support in the form of three engineer-years [...] between the acceptance of PEP 703 and the end of 2025
It seems much more like them pledging engineer time from Meta employees to work on this project than sponsorship, not a monetary sponsorship like the title implies.
"it would be great if Meta or another tech company could spare some engineers with established CPython internals experience to help the core dev team with this work."
Regardless, the previous commenter's point stands that the title could be a lot more informative.
They use Python. They want to donate engineering time. They think this is a good idea. Why does there have to be a sinister motive?
I don't see any charity here, this is just a smart business move by them. When you run as many servers as Meta does being able to run 1/4 as many instances is huge RAM and therefore money saving. It won't take long to add up to 3 engineer/years.
They have the expertise to run a language fork and because of the scale they run at they would easily save more money doing so. Meta contributing to OSS is not an economic decision. It is about branding, trying to attract technically strong individuals (who tend to contribute to OSS or want to), and sticking to their open culture.
Actually, experience has shown the opposite to be true. Time and time again even large organisations find their fork falling behind the community version and the pain of merging new versions from upstream grows and grows
We all remember the anti-OSS 'Linux is only free if your time is free'.
The alternative is Meta puts in no money to salaries and benefits for up to three individual's time on this project, or Meta puts in no money at all.
Meta is paying, one way or another, for this project.
Did I deny this?
To put it in extreme clear example, just said time for someone to punch you != time for someone to work for you, while money is same all across. I hope you don't disagree with it.
You could maybe make the argument that they're trying to swing priorities if this is considered valuable but less valuable than other things, but given the PSF doesn't actually employ anyone who it can direct to work on specific projects anyway, how is this any different from the already existing prioritisation process of "things that somebody is interested enough to go through the PEP process and then work on"?
Their team wants it, and is offering hours to help get it done. Is this not what everyone wants from businesses using OSS?
Philosophically, Python is close to C in the sense that low-abstraction interfaces to system functionality (file descriptors, memory) exist. Numpy allows access to uninitalized memory and you can manipulate how numpy uses memory (mmap, shared memory, sharing memory between objects). Ctypes allows access to malloc and dereferencing dangling pointers. This is not exactly normal Python, but it's inside the realm of the possible in Python.
All of this makes me worried that the challenge of "free threading" is quite large. I wonder if the subinterpreters solution offers a safer route. At least it's not pulling off the GIL band-aid all in one go.
Also a comment on Github is not a binding support contract. Meta executives could deprioritize this project at any time... hell the person making the commitment could already be laid off, we have no clue. As someone who worked in big tech for a long time, trust me - it needs to be in writing with an exec signature.
In this particular case, I understand that removing the Gil would create potential new risks ( but provide better performance) and although tooling is mentioned, it’s not officially part of the plan, so it feels like a regression of some sorts.
That is unless in the serial execution you still have to pay for some resources that are unutilized.
I guess you could say the world went through a dynamically typed loosy goosy phase and then realised that wasn't so great.
I think one thing that actually has changed is that SMT solvers have massively advanced in the last 20 years to the point that formal verification is practical. Ish anyway. It still seems to require a PhD to do software formal verification. Hardware formal verification is easy though.
The FAANG etc failed me for anything open source sponsorship related.
Their business depends on it, they made a fortune out of it, yet they donate little or nothing to it, or worse, they forked it and was reluctant to upstream anything.
pick Amazon randomly, it went from exploiting to rip-off in my opinion, thanks to its boss , who sails on a luxury yacht and asks all his employees to be frugal. they all look similar to me as far as supporting OSS goes.
https://opensource.fb.com/
Next.js exists too which mashed up PHP/Ruby style with React.
It was their authors choice first and foremost to publish them as OSS
There is no legal obligation per-se. Open Source is a charity, but you can understand a lot about a company's motives if you compare their FOSS usage to their FOSS contributions.
Goodwill might work for individual contributors, but it’s not enough of an incentive for organizations.
It _should_ be. There's nothing stopping organizations from doing good things except expectation.
There are real costs and that has real consequences. Expectation is way down on the list.
It’s clearly not perfect calculus since so many defect from OSS, but there’s also clearly some business sense there.
Then it just transitions to a charity that everyone relies on. It's been embarrassingly difficult for people to admit this vis-a-vis Linux and the like, but it's obvious in a situation like programming languages. We rely on the enduring charity of it's authors, license-holders and distributors to write and distribute software.
"Giving back" is not an inherently cynical process. Meta has no obligation to publish these patches in the first place, much less work as a part of the community to solve a common issue. But they do, and it's just kinda part of their culture at this point. Yes, they do still gain from it; but so do their direct competitors. It's difficult to characterize as a purely business-motivated decision, because there are much greedier avenues to pursue if that's their MO.
It is all about knowing individual engineers inside these companies. Nearly any engineer in FAANG can, with a little effort, donate their own work time to an opensource project in the form of submitting patches.
The project can help in various ways. Make sure licensing stuff is in place so that whatever IP review process the company has goes smoothly. Make sure PR's are reviewed fast. Delegate responsibility ('hi, would you like to be maintainer of this feature?'). Make sure contributions are publicly recognised (ie. Public page of maintainers/biggest contributors).
Remember that most engineers inside big companies are trying to generate good content for their performance review. "Wrote and open source some code" is usually one of the checkboxes.
I know at least one of those letters that does not want you doing that without jumping through hoops and getting prior approvals. I will let you guess which one that is.
Meta has its own production fork of CPython, Cinder:
https://github.com/facebookincubator/cinder
There's a whole Silicon Valley episode on this.
Or do license terms apply differently if you’re Meta?
Assuming the company doesn't have clause to not allow contributing to open source. These companies should be publicly shamed if they use any open source code.
I had no problems with this by the way, just saying maybe he can contribute 0.001% of his profit to those OSS projects that he depends on, it will help him to build more yachts too.
One can argue to what extent Bezos is still ultimately "the boss" at Amazon given that he's not the CEO any more, but...
Chromium underpins nearly every major browser that isn't Safari or Firefox. And even more narrowly, V8 powers Node (and Deno).
More Google contributions: Angular, Kubernetes, gRPC, Golang.
Microsoft: C#, TypeScript, Language Server Protocol, VSCode.
Meta: React, zstd, Apache Cassandra, LLAMA.
This isn't even getting into open source patches (Google has consistently been one of the top 10 Linux kernel contributors for the last decade), systems papers that have inspired research and other systems (BigTable, Dynamo, MapReduce, Colossus, Spanner), standards work (HTTP/2 and HTTP/3 were both adapted from Google technology), security vulnerability work (Project Zero).
(The above list is definitely Google-skewed because that's what I know.)
Chrome, though, is a kind of mixed bag. It already has too many features, most likely the reason it underpins so many other browsers is that nobody can keep up with the rate at which Google shovels shit into there.
They should slow down. It would be nice if they could just spend a couple years working purely on stability/performance/security, but alternatively they could just stop, it would be better for the ecosystem.
What FAANG does not do is support critical yet underappreciated Unix infrastructure. By this I mean directly donating to developers without trying to take over the projects.
The quality of FAANG code, especially in the fault-tolerant machine learning space, is questionable as well. I'm not sure why so many commenters (in other comments!) attribute near magical qualities to any output of FAANG.
FAANG are not charities. If they are sold a license for the software for $0, then they don't have much of reason to pay for it.
People apparently forget the amount of money that was and is injected into them.
Someone has to put the money on the table to attend ISO meetings, buy the standards, implement the features into existing C and C++ compilers, for example.
Many of Google's OSS projects are there to help its profits, if not, Google could stop or kill it at will which is fine and is 1 millions times better than closed code, still not the same as projects like python etc.
Do you think they are irrational?
there is graphql, rocksdb and folly. they also open sourced their js engine for react native too.
maybe a good criticism is we depend to much on these companies free stuff lol
I know it is not a popular opinion but the AGPL is there for a reason. If you release your code under a permissive license you should not be surprised that others benefit from it - sometimes massively. That is how it is supposed to be. You can still lament that it is unjust and I feel you, because in a sense it is. If it really bothers you, the solution is easy. Next time choose the AGPL.
Meta and Google have both contributed an enormous amount to open source (creating PyTorch, TensorFlow, Chromium, contributing to clang/LLVM, Linux kernel, HTTP standards), but you're mad because they didn't sponsor your work and a bunch of people mindlessly upvoted it.
“Python 4, at this point whenever it’s mentioned in the core development team, it is very much as a joke… We’ve learned our lesson from Python 3 vs 2, and so it’s almost taboo to talk about a Python 4 in a serious sense.”
[...]
“I normally talk about that as a mistake, because Python was more successful than the core developers realised and so we should have been much more aware and supportive of transitioning from Python 2 to Python 3”
[...]
“I’m not thrilled about the idea of Python 4 and nobody in the core dev team really is – so probably there never will be a 4.0 and we’ll just keep numbering until 3.33, at least”
[...]
“We now have a strict annual release schedule, so after 3.10 will be 3.11 and after that will be 3.12, and so forth. We can go up to 3.99 before we have to add another digit. Adding another digit is not completely trivial, but still much better than going from 3 to 4."
[1]: https://www.youtube.com/live/aYbNh3NS7jA
Most of this work has been on Javascript, which is single-threaded and less mutable than Python, but PyPy has also made some significant contributions. PyPy has a GIL.
There is also work on Ruby. Ruby is even more mutable than Python, but also has a GIL. Work on Ruby mirrors that on Javascript.
Also Carbon is still very much an experiment and not used for much. Google has a lot more Rust in production now, especially in Android and Chrome.
Wouldn’t that still be Python, just a different implementation?
I think the problem there is even a company like Meta wants to rest on the shoulders of the giant Python community. It makes sense for Meta to try to get parts of Cinder into Python.
i really don’t understand the trend with launching a new language every time it seems convenient; it may have the advantage of being made at {company}, but this is also its great downside: it exists, is maintained, and used only with relation to the project/company that prompted its creation
for example, swift is only used for swiftui and some other apple specific stuff, and no sane person is going to write an unrelated serious project in it
Perl never had anywhere near the wealth of libraries and community support Python enjoys today. The language is used nearly everywhere, from utility-scripts to build piplines, desktop applications, webservice backends, all the way up to scientific computing and AI.
Python shines because of it's ecosystem and community support, which I'd guess (without having any numbers to back this up) may very well be unmatched among programming languages today. Maybe Java and C/C++ come somewhat close, but I'd doubt even these behemoths would stand a chance in a 1:1 comparison for sheer size and versatility of what for Python is just a quick `pip install` away.
Any language that want's to replace Python, even internally at a company, will either have to deal with the fallout of leaving all that behind, OR be compatible with Python to an extend that it doesn't matter. And that's a really tall order.
Direct link to the comment, where Meta offers to commit to working on this issue for 3 person-years: https://discuss.python.org/t/a-fast-free-threading-python/27...
The entire debate seems to be for code that strikes me as being far and away from the typical Python programmer.
All the ML code is written in lower level languages and that’s very unlikely to change, GIL or no.
Instagram (and presumably threads) runs on Cinder which is Meta's fork cpython with GIL removed.
I honestly don't think the community is ready for this change and don't expect to ever see stock cpython drop the GIL--perhaps there will be a flag to selectively operate a python process without the GIL (and leave the onous on you to completely test and validate your code works with potentially new and spooky behaviour).
From a python perspective it wouldn't necessarily be a big change, but everything can branch out to c, and there you're going to get in trouble with shared memory.
C extensions have always been able to use real threading. But now there is no GIL to synchronize with the python interpreter on ingress/egress from the extension.
https://realpython.com/python-gil/#what-problem-did-the-gil-...
When I say GIL specific behavior, I mean "python code that specifically requires a GLOBAL interpreter lock to function properly"
Not something that simply requires atomic access or any of the garuntees that the GIL has this far provided, but like, specifically code that requires GIL like behavior above any CPython implementation details that could be implemented with more fine grained concurrency assurances?
I've seen some really cursed python in my days, like checking against `locals()` to see if a variable was defined ala JavaScript's 'foo in window' syntax (but I suppose more portable), but I can't recall anything specifically caring about a global interpreter lock (instead of what the GIL has semantically provided, which is much different)
They are relying on behavior of a single environment. We similarly see a lot of issues moving threaded code to a new operating system in C/C++, because the underlying implementation details (cooperative threading, m:n hybrid threading, cpu-bound scheduling, I/O and signaling behavior) will hide bugs which came from mistakes and assumptions.
Literally speaking, that's exactly what "removal" means. As far as I can tell, GP was wondering why there's so much discussion about replacement, since simply removing the GIL wouldn't break single-threaded code.
Personally I'd much rather see this effort go towards a new language which "feels" like Python but adopts more of the development experience of Go and Rust. From my tinkering it seems like Nim might already be that language, in which case what is needed is investment in its package ecosystem.
The problem with Python 2 to python 3 is that python 3 was essentially a new python 2 esque language instead of just being a major version bump. If python 4 was no GIL python, then both Python and C would remain unchanged as a language.
So, library designers will be under pressure to update their libraries to enable support. We could also have code tools that detect patterns that aren't GIL safe and throw out loud warnings
Python is a single-threaded language. That’s part of its DNA. The community has already been through one traumatic transition in recent history and the appetite for another one is low.
Library authors should not update their libraries to support multithreading, rather the people who want that should be forced to rewrite their code in a language that is more suitable for the problem they want to solve.
Would removing the GIL be a big change for CPython? Yes. But IMO it's worth it
Boooo. Maybe if all you do is ops scripts, but for those of us in data science this couldn't be further from the truth imo.
I'm not saying the syntax is easy for asyncio, or anywhere near as nice as golang or even kotlin is for concurrency, but it's definitely workable in a concurrent environment.
If the changes proposed in this PEP go through, that will no longer be the case. So library authors pretty much will have to either update, or see their modules wither the same way they would if they weren't updated as new Python 3.x versions come along.
> rather the people who want that should be forced to rewrite their code in a language that is more suitable for the problem they want to solve
The vast majority of the Python developer community WANT Python to support true multithreading and being able to solve these problems. We expend inordinate amounts of time and skill trying to make Python work around the GIL, eg. by utilizing multiprocessing.
We want Python to stay relevant long-term. In an age of abundant multi-core platforms, and workloads that can utilize them, the GIL is a major obstacle to that desire.
Err what?
This kind of garbage logic lead to the horrible Ruby/Python/Javascript with C extensions split that makes their ecosystems very brittle, versus Java/C# where it is expected that things are fast enough without C and package management is much easier.
I understand the desire/demand for general purpose tools. The thing is, there are always tradeoffs. Acknowledging the tradeoffs and designing more specialized tools that work well together isn't necessarily garbage logic.
I mean, it's specialized for:
* data science
* AI/ML
* webdev
* DevOps tools
* an embedded extension language for 100000 programs
...
The sooner core Python devs accept this, the sooner the right thing can be done.
Python is not AWK.
Its literally not, though, even with the GIL.
I imagine there would need to be some kind of annotation to enable the GIL for a method and all of the code it calls, including libraries, so performant Python can take advantage of the lack of a GIL but old code doesn't break. Then all you need to do to maintain compatibility is to annotate your main() and your code should remain compatible for a while.
After all, the referenced PEP explicitly calls for making the GIL optional, not for removing it completely.
That wasn't the problem. The problem was not giving people the ability to make their code python 3 compatible while they were still stuck with python 2. The python 3 interpreter should have had a python 2 mode that gives you warnings.
There were compatibility wrappers for porting Python 2 code to Python 3 but in some instances that was just never going to happen.
This does make me wonder, though, if gevent will survive PEP 703's massive changes to internal Python assumptions. That said, gevent does work on pypy, so there's some history with it being flexible enough to port. Hopefully it won't be left behind, otherwise codebases using gevent will see this as a Python 2 -> 3 level apocalypse!
But this only solves network concurrency problems.
Very reasonable tradeoff for an application server, which is embarrassingly parallel and holds little state.
The GIL makes python code have cooperative threading. It does not protect from e.g. your thread's view of state mutating when you make a database call.
I also believe it is best practice not to mutate data without holding the GIL in extension code, not a requirement - but I have mucked with a lot of different extension API so I might be confused.
While C extensions can release the GIL, that only makes sense if they do enough work that a Python thread could get some things done in the meanwhile, and it wouldn’t be surprising to the caller. Obviously the C thread can’t interact with the Python world after the GIL has been released.
Some things really are too big to change.
Then transpile Py3 code into Py4.
And it won't change anything about c extensions.
I'll admit that, yes, changing string to bytes and unicode to string was a bit annoying, but the change itself wasn't fundamentally 'of dubious benefit', it did have benefits, and related to this, the only major issue was that you couldn't, for a long time, have code that worked in both where it came to literals. The biggest problem here was the implicit conversion from 2, that I agree needed to go.
Most of the other things can be trivially fixed automatically, or at least detected automatically, but without type hinting, it wasn't really easy to fix the automatic conversion.
There were other changes that were a bit tricky, but the majority of issues stemmed from the str/bytes change.
Worked for ruby. The original interpreter MRI has a GIL too. Rubinius and JRuby added multi-threading with limited amounts of pain and people fixed libraries over the years. Sometimes just sprinkling lock blocks around a particular FFI calls or only doing them from a dedicated thread will do the job.
There are a lot of things to be worried about with the GIL conversion of new race conditions that could happen, but there's already too much misinformation out there about the GIL, let's not spread this one even further.
My guess is the GIL or smaller mutexes will be needed for C extensions and a few other areas, but it’s also likely that could be moved to an opt in mechanism over time.
Not really, just make those operations atomic or have automatic locking
I'm sure they'd welcome a patch to fix the forums.
https://discuss.python.org/t/a-fast-free-threading-python/27...
Should take you to the first post, which is an overview of the proposal problem and potential solutions.
Mobile Chrome renders the page just fine. No idea about what other WebView you're using, but JS definitely needs to be enabled to load additional content.
In short, GIL is the global interpreter lock, something that is innocuous in single threaded software but a bane for performance in a true multithreading environment. I'd recommend reading up on it elsewhere if you're not familiar with it, as the post elides a lot of detail (the audience is the forum is, after all, people already familiar with python internals).
"nogil", therefore, is a shorthand for CPython running without the global interpreter lock. The crux of the proposal is a discussion on how to achieve it without making large sacrifices to reliability or performance in single threaded scenarios.
> No idea about what other WebView you're using,
Android default, so also chrome-like, just not the proprietary Google Chrome browser
> "nogil", therefore, is a shorthand for CPython running without the global interpreter lock.
Ooh, I should have realised what nogil means. If this had been in the headline with capitalisation like noGIL, I wouldn't even have needed to click to understand what it is Facebook wants to work on. Thanks!
I'd wager the vast majority of programs being written that need concurrency or parallelism still use multi threading or SIMD.
Both things aren't exchangeable, of course!
Thread have direct access to process memory. Other processes usually don't. Big difference!
Meanwhile, the Steering Council is dragging their feet, even though so many people in the community have chimed in their support for nogil being a thing (in the linked thread and two others).
https://github.com/python/steering-council/issues/188#issuec...
> The PEP was posted five months ago, and it has been 20 months since an end-to-end working implementation (that works with a large number of extensions) was discussed on python-dev.
I would expect the SC to be active participants in the community, including the python-dev mailing list, so they should have known what this change means by that point.
Most of the time in an interpreter you’re iterating through the byte code for a function, and mutating variables. It doesn’t matter if two threads are executing that same byte code, it only matters if they are mutating the same object.
That’s not to say there aren’t some operations that need coordination across all threads. For that you implement safe points so you signal that something has to be done, wait for all threads to reach a safe point, do the work, and then release the threads again. You can often be even smarter about this and avoid stopping all threads at once but instead stagger things.
Allocation and GC you can make more thread local to avoid stopping the world too often.
Without the GIL, you'd still need to use different locking/synchronization mechanisms to ensure thread safety (if required), but they'd be more nuanced and spread over the entire codebase (complicating the backend).
I know Clojure is often described as a dynamic language and that it has some concurrency primitives, but my knowledge ends here.
[1] Just like Python, Ruby does have implementations that allow multi-threading.
This comment is the definition of FUD.
The issue is all the libraries and packages that were built around it. It's now been YEARS of building on top of the GIL.
So anyways, PEP 703 is a a nice effort, but I doubt we (everyday mortals) can enjoy it. Meta, and big companies with a specialized team might be able to exploit it by making sure their entire stack is GIL-free.
EDIT: My bad, the talk is from Larry Hastings: https://www.youtube.com/watch?v=P3AyI_u66Bw
I'd suggest we can be more optimistic. There is a lot of python, regular users write daily, where we delegate orchestration to established libraries - asyncio/web-frameworks/pytorch. The GIL limits how much they can parallelize your code, and its removal will help with that.
Not saying he isn't wrong, though.
Not really. Most of the popular libraries will have nogil versions ready for sure (numpy, pandas, pytorch, etc). Server applications will probably benefit from being able to serve multiple requests in a real multithreaded environment without many changes (in fact, I bet a lot of HTTP libraries will see simplifications due to not needing to deal with processes for real parallelism). Even packages without changes will probably have at least a small degree of thread safety (for example, two instances of the same object should be independent and could be used in different threads) that allows users to still leverage parallelism.