This is not unique to Python-pip. A bunch of popular packages are maintained by very few people. But the good thing is that it does not really matter than much with FOSS projects: as long as the source remains available, anyone can pick it up at some point and improve on it - or you can even contract people to work on it when needed to.
There's a fine line between "there are so few maintainers that nobody could spare the attention for this" and "there are so many maintainers that this slipped through the cracks".
Really, it's less of a line than a region of negative width. Number of maintainers is not a good indicator for the security position of the project.
Very open - Python is one of the better communities out there for openness and inclusion. The pip folks are lovely people and would welcome your help - https://pip.pypa.io/en/latest/development/.
I can't speak for PIP specifically, but something I've noticed in Python (just bc that's where I work) is that lots of projects would greatly benefit from more handholding for new contributors.
Obviously it's a question of limited resources, but I wonder why projects don't offer a clear path to growing as a contributor. For example, once you triage 3 tickets and close 5 documentation tix, you get assigned a mentor and a few features specifically set aside for mentor+mentee to build together. Then use these not as methods of gatekeeping (let anyone contribute as much as they want), but as a way to pull in a few new people at a time.
I also think that many projects would benefit from having formal "apprentices". Being an apprentice for React would be an amazing thing to have on your resume, and would cost no money for the project.
The longer I've been around open source, the less it's seemed like some ideological adventure where we extend human knowledge and capabilities for its own sake, while looking more and more like a set of intentionally confusing layers of abstraction and indirection around companies with plenty of money to spend on more engineering hours contriving to buy those hours well at below minimum wage by selling people on the above ideological vision.
Also, I'm going to guess that's something like five active contributors signed up, not five full-time engineers, meaning the actual number once you do the math for availability and such is probably closer to half an engineer. Could be five, but Heartbleed kinda put paid to reasoning like "pip is important, surely it has the funding for dedicated contributors who aren't distracted by something else being their real job."
Money is always decoupled from value ( Warren Buffet paraphrased ).
Sometimes people who add negative value like the oil men in the Permian basin get paid 100x then your pip developer.
Screw that, large amount of the surplus generated by technology companies ends up in the hand of land owners instead of stockholders ! let alone the engineers building it - who might be riding on the work of open source volunteers. Its turtle all the way down.
If you are a volunteer, you are forgoing payment and are subjecting yourself to exploitation by others by definition.
Unless you have a mechanism for enforcement you are shit out of luck.
Isn't this the point of "copyleft"? Create a valuable product, require that when someone builds on it, that what they create also be free to use.
Open source software fails like this even if it's valuable because fixes and improvements don't necessarily make it out into the open, whereas Free/libre software, if it's valuable, gets stronger over time because fixes and improvements must be available to everyone.
Adding "fixes and improvements" to software haphazardly does not, in general, make it stronger. Quite often it can be the opposite.
Software projects need someone to maintain its course, make sure changes match the purpose of the software, and most importantly, reject changes that create a burden on the project.
The point of copyleft is more ideological; it prevents others from using your software to restrict user freedoms, thereby limiting the spread of harmful, nonfree software. Copyleft is primarily intended to protect users.
For me it's both. I'm not going to work on a project for the sake of open source, but ideology is a big part of why I choose to release my code under an open source license. In fact, if I need something that I think would be useful to the community (or myself on other projects), I'll build it on my own time and then do bugfixes and whatnot on company time.
People are all over the spectrum, with people like Stallman and developers at the FSF working mostly on ideological grounds and companies like Oracle only contributing when it would clearly be beneficial to the bottom line.
I used to think I would love to work on OSS if I didn't need money, but I've found that I just like working on stuff, and OSS isn't as much of a priority anymore.
Although I think five may be quite good comparatively, my thanks to the pip developers.
And if five is not a good number, this is indirectly one of the reasons that I enjoy using python. It has a 'batteries included' release, where I often write code without feeling like I need to $ pip install anything at all. If the 3rd party package delivery system is fragile, that isn't good - but it doesn't represent a dire crisis for me as a user of python.
Personally I'm more concerned about the attitude and atmosphere that implies that 5 maintainers are too few for a mature project with relatively limited scope. I'm more and more beginning to think how can we reduce the ridiculous amount of useless churn in software.
The problem is exactly that pip is neither mature nor has limited scope. Python packaging is constantly evolving, and pip needs to keep up with it. And you’d be very surprised if you ever look under the hood how incomplete the implementation actually is.
At Google there is an internal document called "No Heroes". It basically says that if your load is too high - let the damn thing fail. Perhaps it's not important, and only you think it is. Perhaps the higher ups don't realize it's important and they need to be reminded of it. Reliable infrastructure cannot depend on heroic actions of a small group of people, and especially on actions of a single person. Let it fail - the world will take notice. Or not, in which case you should move to something more useful.
Not affiliated any more, so I don't even have it. But yeah documents you write on payroll are corporate intellectual property (same as code is) so I couldn't just share it even if I wrote it myself.
I wish they added it to the SRE book. But it's not a hard sell if someone already believes that you need to distribute computation/data for higher reliability. Same goes for labor/know-how.
> It basically says that if your load is too high - let the damn thing fail. Perhaps it's not important, and only you think it is. Perhaps the higher ups don't realize it's important and they need to be reminded of it.
I do have friends at google who can get away with this approach, but have more who feel they would be fired for doing this.
Wow! Five people dedicate time to maintain pip for no profit!
It's pretty amazing like the post states. After openssl's heartbleed I remember hearing about a critical FOSS fund to help with things like this. Much like how wikipedia is funded,it makes sense to fund critical projects like pip and npm.
Most commenters are rightly worried about the “bus factor” element, but I’m just amazed at the returns that you can generate these days with so little manpower.
Five people literally enable entire ecosystems of developers across the world, multiplying their productivity by ungodly amounts and indirectly generating billions of dollars of value. Isn’t that extraordinary?
52 comments
[ 290 ms ] story [ 2559 ms ] threadhttps://github.com/rust-lang/cargo/graphs/contributors
https://github.com/rust-lang/cargo/graphs/contributors
Maybe 2/3 people actively doing most of the contributions with sporadic other contributions.
Really, it's less of a line than a region of negative width. Number of maintainers is not a good indicator for the security position of the project.
https://en.wikipedia.org/wiki/Bus_factor
Obviously it's a question of limited resources, but I wonder why projects don't offer a clear path to growing as a contributor. For example, once you triage 3 tickets and close 5 documentation tix, you get assigned a mentor and a few features specifically set aside for mentor+mentee to build together. Then use these not as methods of gatekeeping (let anyone contribute as much as they want), but as a way to pull in a few new people at a time.
I also think that many projects would benefit from having formal "apprentices". Being an apprentice for React would be an amazing thing to have on your resume, and would cost no money for the project.
Besides that though, I wondered how much maintenance pip even needs. Shouldn't it be kinda stable?
But looking at https://github.com/pypa/pip/graphs/commit-activity it doesn't seem so.
Also, I'm going to guess that's something like five active contributors signed up, not five full-time engineers, meaning the actual number once you do the math for availability and such is probably closer to half an engineer. Could be five, but Heartbleed kinda put paid to reasoning like "pip is important, surely it has the funding for dedicated contributors who aren't distracted by something else being their real job."
Money is always decoupled from value ( Warren Buffet paraphrased ).
Sometimes people who add negative value like the oil men in the Permian basin get paid 100x then your pip developer.
Screw that, large amount of the surplus generated by technology companies ends up in the hand of land owners instead of stockholders ! let alone the engineers building it - who might be riding on the work of open source volunteers. Its turtle all the way down.
If you are a volunteer, you are forgoing payment and are subjecting yourself to exploitation by others by definition.
Unless you have a mechanism for enforcement you are shit out of luck.
Open source software fails like this even if it's valuable because fixes and improvements don't necessarily make it out into the open, whereas Free/libre software, if it's valuable, gets stronger over time because fixes and improvements must be available to everyone.
Software projects need someone to maintain its course, make sure changes match the purpose of the software, and most importantly, reject changes that create a burden on the project.
People are all over the spectrum, with people like Stallman and developers at the FSF working mostly on ideological grounds and companies like Oracle only contributing when it would clearly be beneficial to the bottom line.
I used to think I would love to work on OSS if I didn't need money, but I've found that I just like working on stuff, and OSS isn't as much of a priority anymore.
And if five is not a good number, this is indirectly one of the reasons that I enjoy using python. It has a 'batteries included' release, where I often write code without feeling like I need to $ pip install anything at all. If the 3rd party package delivery system is fragile, that isn't good - but it doesn't represent a dire crisis for me as a user of python.
I wind up having to install as many packages as dealing with a Node JS project
Let's nominate them for PSF fellowship https://wiki.python.org/moin/PythonSoftwareFoundation/Fellow...
I think there is a reasonable argument that that has been part of the problem with Python packaging.
If it is at all possible to post it somewhere?
Is it breaking any obvious NDA or can you share it?
I wish they added it to the SRE book. But it's not a hard sell if someone already believes that you need to distribute computation/data for higher reliability. Same goes for labor/know-how.
I do have friends at google who can get away with this approach, but have more who feel they would be fired for doing this.
They're not getting away with it. That's the way it be.
Your other friends need to take away the "internal document" part from this story not the "Let things fail" part. They're not there yet.
> 2-3 devs at 5 hours per week. [0]
The Ruby Together initiative aims to make the situation better through recurring donations.
[0]: https://rubytogether.org/
It's pretty amazing like the post states. After openssl's heartbleed I remember hearing about a critical FOSS fund to help with things like this. Much like how wikipedia is funded,it makes sense to fund critical projects like pip and npm.
It's owned by a private entity and it's server side code isn't available to my knowledge.
Five people literally enable entire ecosystems of developers across the world, multiplying their productivity by ungodly amounts and indirectly generating billions of dollars of value. Isn’t that extraordinary?
It's even worse for pipenv. It was last released in November last year and there are a lot of bugs fixed on master, but there is never a new release.