Very interesting and I hope this leads to actual recompense/compensation for people "thanklessly maintaining" OSS. Not sure how it possibly fits into Google's newfound profit-at-all-costs motive, but the gesture is an important first step towards some semblance of justice.
Can you elaborate more on the claimed "profit-at-all-costs" statement? It's been established that the purpose of most corporations are for profit, but iirc most large enterprises have a rather robust ethics framework when working with the OSS community.
I wish I had reasonable evidence to point to, but I'm not really an insider. Mainly, I think of the numerous applications Google invested lots of engineer time into to create an experience, say Hangouts, only to realize that it hampered sales in another corner of Google, say cell plans, and so one project must be discarded for profits in the other. The functionality is often difficult to replace for the end-user, and weakens the overall allure of the Google network/ecosystem of applications in favor of a bottom-line.
Sounds interesting in theory, but the results are somewhat bizarre. For example, their published list of the top 200 "most critical" open-source C projects includes Urbit and Stellar, and includes micropython but does not include CPython.
As per Github (see languages section on bottom right), Cpython is 63% Python, and 29% C code, so its api returns Python as main language. cpython is in top 10 in python_top_200.csv
Seems like a good direction. My main hope is that if metrics like these become more accepted and/or important, that they find a way to avoid running afoul of Goodhart's law.
Tensorflow ranking higher than Linux is a weird one. Tensorflow debuted to a huge amount of hype, but it could fall over and nearly all our software systems would be totally fine.
World is not perfect. In case of gcc, it is only a mirror on github [https://github.com/gcc-mirror/gcc], so we dont get all the metrics, we do plan to improve this part [but this will be slow to clone repo, know their custom issue trackers, etc].
Gnucash is still a popular project, has like ~7k downloads a week, see wikipedia page.
Strangely enough that's not mentioned anywhere on the announcement. In fact, it seems to announce that very real funding decisions are going to be made based on this score already.
How can you make accurate decisions based on inaccurate data?
Which reminds me of that Babbage quote: "On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."
> Gnucash is still a popular project, has like ~7k downloads a week, see wikipedia page.
omg they're actually doubling down on this. Libreoffice has something like 50k downloads per day and isn't listed, while among "critical projects" there is... minetest ? tesseract ? battle for wesnoth ? dolphin ? citra ?
There's qt creator (and a ton of Qt-using projects) but not Qt ?
Maven and Gradle both missing from top 20 on Java.
Whatever one thinks about Java development, if those two tools disappeared most Java development world wide would stop.
There would always be the lone developers deploying from IntelliJ/NetBeans/Eclipse and a few people using Ant/Bazel/Ivy but I must admit I haven't seen any of those "in the wild" the last three years.
Jup. Amongst the 200 "most critical python libriaries" are for instance https://github.com/plamere/spotipy, a client API for Spotify. I don't believe that the majority of Python codes want to play spotify streams :-)
The criteria https://github.com/ossf/criticality_score/blob/main/README.m... are designed to give higher scores to projects that are updated frequently by many contributors from different organizations, so of course the random Nebraskan's critical project is going to get a low "criticality score".
I'm not sure if this is relevant, but bash and the readline lib are both maintained by a single unpaid volunteer. (I don't know if he's from Nebraska though.)
Gensim is #119 in the list according to your link, far behind projects with many more active contributors, so hardly a resounding success of your scoring method.
In terms of metrics, you could start by weighing projects with few contributors as more critical, not less. Specifically, gensim does appear to have had quite a few contributors, but the bulk of the code was written by the single maintainer https://github.com/RaRe-Technologies/gensim/graphs/contribut... So maybe you should add a metric "percentage of code in the past year authored by the top contributor".
If you want to go about it in a more data-driven fashion, you could go through the top projects for each language, check whether they actually need your support (e.g. find out what the development goals are, ask whether the current resources are sufficient and what they'd do with the additional resources you can provide) to get a ground-truth labeling of critical projects, then readjust your weights to match the ground truth.
I think we could improve it a bit. For example, Spring boot should have a very low score for me. It's backed by a large company Pivotal. They don't need any support I think. Same thing for elasticsearch.
For me:
- backed by a large company ?
- number of contributor doing 80% of the work ? or active in the last 12 months ? commits breakdown (99% is done by one guy) ?
- issues created/closed ratio
- PR created/merged ratio
- use critical projects ?
- other from your original score
A nice bonus: if we could use the tool to assess critical score for our project (not globally). For local dependency, we could increase the critical value if dependents count is low. Very few person is using it: that's a bad sign. With this, we could find those dependencies.
We could also create a global score (like you did) by using the previous score and scaling it using the dependency usage (dependents_count like you did).
With this calculation, I think it's more likely to find relevant projects.
How to find it's backed by a large company ? Not sure about this, we can check if the project is part of an organization, if contributor have a company or if they have a pro account. For example, if the top 5 contributors are from Google, it's likely it's sponsored by it(could be done during their free time but less likely).
Note: check what happens with a stable project (no new issue and PR).
I don't think it's explicit in the article, but that's a comic (XKCD). Relevant discussion [1] suggests that there's not a specific project referenced by the comic.
When I first read this comic, ntpd [2] [3] came to mind.
At least NTP is a standard protocol and there are a bunch of cromulent alternatives to the original Mills ntpd. Chrony and ntpsec are both reasonable. If any one of the implementations went away tomorrow, distros would scramble but they'd have a place to go.
I'm interested in the gulf between low-level systems code and effectively end-user code here. Things like GLib, Cairo, Harbuzz, etc -- none of the end-user tools work without them. It really demonstrates just how difficult this evaluation can be.
It is definitely difficult especially with critical dependencies. We are looking for any criteria to identify these in automated fashion. Parallely, for ones we know about those are we are trying to run our automated tools on. E.g. glib, cairo, harfbuzz are all continuously fuzzed as part of OSS-Fuzz - https://github.com/google/oss-fuzz/tree/master/projects
The issue with the mirror is we don't get the important stats to make decisions. E.g. number of contributors, issue changes due to custom issue tracker. We are still thinking on how to add information from such cases in automated fashion, ideas welcome!
Please consider allowing scanning tarball/zip distributions of source directly as well. It is a SCM-agnostic method that is also well-supported by GitHub, Gitiles, hgweb, and many old but still-in-use projects that pre-date Git.
It would be nice if this could be mentioned a bit clearer in the blog post and/or README; it's not really that obvious at all and I had to go to the source to check, and loads of people here seem confused about it since it more or less implies "we looked at all open source projects".
That algorithm seems unnecessarily complicated and includes somewhat dubious metrics when, in my mind, the only thing that really "counts" when it comes to "criticality" are "how many other things use/depend on me", and there are much easier ways to determine this:
1. For languages with a common package repo, how many other packages depend on me? With NPM, for example, it's pretty easy to figure out how many other packages depend on a given package.
2. For "top level" projects, look at downloads.
My guess is just looking at either (or both) of those metrics would give you better results.
- Other metrics such as how many contributors and organizations are involved, how many user feature requests and bugs getting reported, those are all important project importance and not just "dependency count". some projects can be standalone, so as per your algo, those should be very low.
- Downloads data is not available for most repos, please find a reliable metric to use.
- Package repo dependencies works, but it does not work for C/C++. At some point, we will integrate github's dependency count info as well.
So basically the algorithm is designed to not find projects that are critical because they are deep in the foundations, depended on by nearly everyone but only worked on by a few people? (which is what "critical" would suggest at least to me) This seems to be mostly a "github marketing index"...
I seriously hope no actual decisions about resource allocation etc are made based on this.
We are working on this problem, it is not simple. Identifying dependency trees reliably across languages is not straightforward [only nice for package manager ones]. Follow https://github.com/ossf/criticality_score/issues/8
You didn’t identify certifi, urllib3, chardet or pytz in your top 10 critical Python dependencies. These are all highly download packages, mostly maintained by one person, which are totally critical to millions of other packages and the Python ecosystem as a whole.
A few of your top-10 I can agree with, but when you’re saying a home-automation package (“core”) is
more critical than something like pytz then something has gone terribly wrong.
Github's downloads don't tell you when FAANG added your project to their internal project tree, which is then deployed in hundreds of millions of devices.
The methodology is pretty silly. It rewards activity and popularity.
A lot of critical infrastructure software is not active and not often spoken about!
Case in point: The https://www.cip-project.org/faq project highlight the needs of very-long-term support for OS components that run on critical infrastructure such as power stations.
We have to start somewhere. It is understandable that this is not complete, so welcome your ideas to discover such projects. Please think of any metrics/ways to find such projects.
I strongly recommend you use the packages maintained in Linux distributions as a means for discovery. They're well-organized and maintained and easily accessible programmatically - you can even parse the package dependencies programmatically, as well as have full access to the original source code.
good advice. "apt-get install apt-rdepends" and it becomes possible to work out the reverse-dependencies of packages.
by counting the numbers it becomes pretty blindingly obvious what the critical dependencies are. as mentioned in another post above, bash and glibc6 are blindingly-obviously high on the list... yet the GNU Project receives an unbelievably low amount of funding despite their critical importance.
likewise, this particular bug in binutils ld, which centres around the incredibly short-sighted "4GB should be enough for anyone" removal of Dr Stallman's memory-resident algorithms in the late 90s, is having some very serious consequences:
yet because there's no money not even from redhat nobody's looking at it.
likewise: PAM no longer has a proper maintainer, and hasn't had for... a decade?
these are projects that people are relying on yet completely forgetting they're a critical part of the infrastructure!
why? because, just as rhencke said above: they're not on github, they've not got "unnecessary changes" which are counted as "activity to be glorified and worshipped".
abharya: i heard on slashdot the intent to start from github, to exclusively focus on github. this will turn out to be a serious mistake.
There is no exclusive focus, we are just starting somewhere where we can see the various metrics. Plan is to expand to non-github projects and other places (like custom issue trackers), but this is not straightforward as it sounds. Ideas welcome!. https://github.com/ossf/criticality_score/issues/29
I would like to see a measure of criticality that takes the following into account:
* Critical projects may have very little activity/maintenance. For example, Bash 4.0 to Bash 5.0 was only 123 commits over 8 years. But, Bash is a absolutely a critical project (ask any org about how much work they had to do when affected by https://en.wikipedia.org/wiki/Shellshock_(software_bug) ).
* A measure of criticality should understand _as many of the various forms of dependence on software_ that may occur that it can. Dependencies can take many forms, such as:
a package manager resolving a dependency
a user purchasing a mobile phone with software pre-installed
a user visiting a website (react/jquery/etc)
etc.
* Criticality should understand if, how, and when dependencies are updated. For example, fixing a bug in Chrome and distributing that fix to 80% of users in 1 week is feasible. Fixing a bug in Bash and distributing that fix to 80% of users in 1 week is not so feasible.
one of the things that's important is not to include google's lawyers bias against the GPL license. i always wondered where the bias against the GPL came from, within google, and learned of the existence of the in-house legal team. it turns out that they have been advising google employees for some considerable time, "avoid the GPL, avoid the GPL".
unfortunately, as legal advice, those google employees (right the way to management) do not have the backbone to say, "err no actually, GPL-licensed code is the critically strategically important leveller that forces aberrant companies to collaborate rather than sponge off of underfunded projects".
people. as a computer scientist you're probably thinking, "this can be solved by analysing a source code forge" or, "this can be solved by running an algorithm". it can't (or, more to the point: it can tell you quantities, but not quality or value).
i mentioned in another post: github "glorifies" the person and the changes that they make. "look at mee! look at mee! i'm making a commit! i'm wiping my backside now! aren't i great!" which gets you precisely zip in terms of actual strategic value.
changes measure change.
people will tell you - if you let them - by providing you with the information needed to make a qualitative assessment.
so.
provide a type of wiki/website that allows qualitative assessments to be made, on a per-library / per-project basis. then put the metrics (the "criticality value") onto that.
pre-seed that wiki/website with stuff from github if you feel so inclined but DO NOT limit the wiki/website to exclusively github. i repeat again: doing so would be a disastrous mistake.
The activity metric is especially frustrating because it seems to me that the higher quality a project is, the less bugs will be filed, and so the less activity will result.
With one of my open source projects I've got to extreme lengths to test it under a huge number of compiler configurations, so it compiles on anything with no warnings. I get nearly zero bugs filed when people port it to esoteric platforms.
When randomly searching a while back, I found a blog post someone wrote about getting my project running on Arduino [1], something I had never tried. Turns out they just had to set some configuration flags and everything worked fine. They therefore filed zero bugs, and it resulted in zero activity for my project. They didn't even tell me about it!
I was extremely pleased that it worked of course, but disappointed that my project continues to look like it's dead. I imagine Google's criticality score for my project is near zero.
One cool feature of functional package managers such as Nix and Guix is that the dependency graphs are entirely transparent and can be inspected programmatically.
I wrote a script that lists the number of dependents for each package in Guix by traversing the package graphs:
Guix and Nix _are_ source-based distros. They just happen to have a facility for transparently downloading "binary substitutes" for local builds.
The graphs I'm traversing are the build-time dependency graph of all packages. I could also traverse run-time dependency graphs, but then I'd need to actually build all the things, because those are not known up-front unlike Gentoo and Macports.
The reason the toolchain is missing is because it is an "implicit input" as opposed to a "normal" input, so it's invisible/transparent to the query I'm making.
Maybe I mis-understood what's supposed to be being checked here. I thought the point was surfacing critical projects that are under supported.
So a highly dependent project that has plenty of support for its needs should not need to be brought to anyone's attention. I'm guessing python, llvm, clang, chromium, webkit, C#, visual studio code, off the top of my head are all well funded and supported open source projects.
I don't know what good examples of under supported but critical open source projects are. I guess I've read that OpenSSL was massively under supported but apparently the solution chosen was to throw it under the bus and promote libssl or boringssl or something like that (completely out of my expertise)
It does seem to be popular w.r.t user downloads and other github metrics.
E.g. https://sourceforge.net/projects/gnucash/files/stats/timelin...
Wikipedia - "As of July 2018, SourceForge shows a count of over 6.3 million downloads of the stable releases starting from November 1999[24] Also, Sourceforge shows that current downloads are running at ~7,000 per week.[25] This does not include other software download sites as well as Linux distributions that provide download from their own repositories."
> This does not include other software download sites as well as Linux distributions that provide download from their own repositories.
Isn't that a pretty significant difficulty for the interpretation of this number, though? Couldn't there be some package that is installed by default in, say, Ubuntu (and that Ubuntu can't boot without), so therefore millions of users are using it -- but perhaps the "downloads" seen by GitHub or SourceForge are mostly by OS packagers or by developers contributing to that package. Whereas if there is something where, for some reason, end-users customarily or commonly get it from GitHub or Sourceforge, it might look more widely used even though it is actually less used overall, potentially even by orders of magnitude.
(Like GNU coreutils, at an extreme, is probably not that popular at all for end users to download from a source control system, yet it might be running in tens or hundreds of millions of devices, which could not even boot without it.)
There will always be edge cases and scenarios we are not taking into account, please provide feedback on issue tracker and provide any suggestions so we can account these.
I mean, our company uses elasticsearch. But like, if it went away, it would not be critical or something. Some features would degrade a bit and no one would cared.
But yeah, the list is bizarre. It is more of "list of cool things to read about that you dont really need" then "critical list".
For example, it counts the number of issues as determining criticality. If there is any kind of money attached to this score (now or in the future), obviously this is going to encourage people to introduce more bugs into their projects.
It's nice that they're automating this even though the results seem currently a bit random. Fedora has a concept of Critical Path packages (https://fedoraproject.org/wiki/Critical_path_package). The concept is related to various activities, eg. a package is critical if is is needed when installing a graphical desktop. But as far as I know packages are picked manually and therefore we probably miss packages or over/underestimate criticality.
I like this idea, which pops up here and there occasionally, but this particular "criticality score" appears to measure popularity, rather than criticality. For one, it doesn't take into account whether there are common alternatives to a given project/package. Of course data like that is hard to gather automatically and from GitHub, but possibly that's just not a great source, and/or could be combined with others -- such as operating systems' repositories, which would contain information not just about alternatives, but also about packages required to run a minimal system, and to build it.
Have you heard of the WWII bullet holes problem that Wald solved? This project seems to fall into the trap that Wald’s colleagues fell into. The more popular a project the more critical it becomes when that’s not actually the case. You want to know where the bullet holes you don’t see are because that’s the thing that’s going to take you out. All the bullet holes you’re seeing and surviving are not great but survivable.
Critical to me means “what’s the thing I have to pay attention to or I’ll suffer consequences”. Having a dependency on a non-popular crate is definitely an increasing score of criticality for that dependency. I may have misread but the fatal error in the metric to me is that popularity of a project increases its criticality when it should decrease. A popular project with lots of contributors means it has lots of stake holders already and a way to successfully manage that. What you want is the small independent projects that popular projects depend on. In other words, what’s the smallest, hardest to notice malicious change I can make in the supply chain of software development to enact the most disruptive change?
This is still an early project to start somewhere.
We are working on this problem as well. Trying to identify dependency trees and which smaller projects are the most widely used (indirect deps) and will impact the most on the critical projects in the list.
Don’t get me wrong. I appreciate this work and it’s super important if you get it right. I’m worried the first step is drastically in the wrong philosophical direction as far as the metric itself goes.
If you’re just proving out the tech then do that but the current publishing of the most “critical” projects is laughably wrong (as I’m sure you’ve heard from the comments on the page). The project isn’t a dead end but any attempt to use the data right now to drive decisions is a lost cause. Maybe you have some secret sauce that makes it work better for internal Google projects but if that’s the case that methodology should be published as well so that people can get an understanding of how Google uses this more effectively.
I’m just contrasting this to a paper Google published recently about how to measure uptime more effectively in cloud. Every step of that paper was eminently approachable and didn’t have any obvious philosophical flaws that stood out to me (we can argue about them but any potential arguments were called out).
Again, I applaud the attempt. This is a super important problem and trying to tackle it is laudable. Getting the right metrics is crucial and this is being released far too early with a presentation that makes it seem far more complete than it is.
what i strongly recommend that google do - instead of doing this work which is, as you've probably noticed from the comments, well-meaning but highly likely to be biased - is:
*make a large donation to NLnet*
don't tell them what to do with the money: leave that up to them. NLnet is extremely good at ensuring that money is used effectively, by requiring that they come up with a project plan involving milestones. they do not just "dump money at the developer", which has a known high historically-backed probability of causing more harm than good: they require the milestones to be completed, 100%, before the money is paid.
note, here: they do not use algorithms to assess a project: they use people. they also assess the proposal against the usefulness of achieving key objectives.
101 comments
[ 4.4 ms ] story [ 164 ms ] thread- Node (0.984)
- Tensorflow (0.969)
- Git (0.945)
- Linux (0.936)
- PHP (0.919)
The rest in that list seem reasonable.
https://commondatastorage.googleapis.com/ossf-criticality-sc...
(from the repo)
So gnucash is #15 . At #75 is gcc .
This seems like a great idea, but perhaps some refinements are needed.
Gnucash is still a popular project, has like ~7k downloads a week, see wikipedia page.
How can you make accurate decisions based on inaccurate data?
Which reminds me of that Babbage quote: "On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."
omg they're actually doubling down on this. Libreoffice has something like 50k downloads per day and isn't listed, while among "critical projects" there is... minetest ? tesseract ? battle for wesnoth ? dolphin ? citra ?
There's qt creator (and a ton of Qt-using projects) but not Qt ?
And we do agree that there are several critical packages missing that don't use all metrics on github, plus non-github projects. Capturing this in https://github.com/ossf/criticality_score/issues/21
Whatever one thinks about Java development, if those two tools disappeared most Java development world wide would stop.
There would always be the lone developers deploying from IntelliJ/NetBeans/Eclipse and a few people using Ant/Bazel/Ivy but I must admit I haven't seen any of those "in the wild" the last three years.
- Python: salt, core (https://github.com/home-assistant/core), pandas, scikit-learn, numpy, airflow, erpnext, matplotlib, pytest & pip
- Rust: servo, cargo, rust-clippy, tokio, rust-analyzer, tock, tikv, alacritty, libc & substrate
- JS: node, react-native, react, gatsby, three.js, bootstrap, material-ui, odoo, next.js & Rocket.Chat
- Java: elasticsearch, flink, spring-boot, hadoop, netty, jenkins, beam, bazel, alluxio & pmd
- C++: tensorflow, ceph, pytorch, bitcoin, electron, Marlin, Cataclysm-DDA, llvm-project, rocksdb & QGIS
- C: git, linux, linux, php-src, openssl, systemd, curl, u-boot, qemu & mbed-os
...which is the exact opposite of what it should be.
In terms of metrics, you could start by weighing projects with few contributors as more critical, not less. Specifically, gensim does appear to have had quite a few contributors, but the bulk of the code was written by the single maintainer https://github.com/RaRe-Technologies/gensim/graphs/contribut... So maybe you should add a metric "percentage of code in the past year authored by the top contributor".
If you want to go about it in a more data-driven fashion, you could go through the top projects for each language, check whether they actually need your support (e.g. find out what the development goals are, ask whether the current resources are sufficient and what they'd do with the additional resources you can provide) to get a ground-truth labeling of critical projects, then readjust your weights to match the ground truth.
I think we could improve it a bit. For example, Spring boot should have a very low score for me. It's backed by a large company Pivotal. They don't need any support I think. Same thing for elasticsearch.
For me:
A nice bonus: if we could use the tool to assess critical score for our project (not globally). For local dependency, we could increase the critical value if dependents count is low. Very few person is using it: that's a bad sign. With this, we could find those dependencies.We could also create a global score (like you did) by using the previous score and scaling it using the dependency usage (dependents_count like you did).
With this calculation, I think it's more likely to find relevant projects.
How to find it's backed by a large company ? Not sure about this, we can check if the project is part of an organization, if contributor have a company or if they have a pro account. For example, if the top 5 contributors are from Google, it's likely it's sponsored by it(could be done during their free time but less likely).
Note: check what happens with a stable project (no new issue and PR).
When I first read this comic, ntpd [2] [3] came to mind.
[1] https://www.explainxkcd.com/wiki/index.php/2347:_Dependency#...
[2] https://lwn.net/Articles/701222/
[3] https://lwn.net/Articles/713901/
The main Go SQLite3 for accessing SQLite databases is in the top "C" list.
But SQLite itself doesn't seem to be included in any of them. o_O
https://github.com/sqlite/sqlite/
This probably omits some other projects as well which don't use git or GitHub.
https://github.com/sqlite/sqlite/
cplusplus_top_200.csv:llvm-project,https://github.com/llvm/llvm-project,C++,48,0,2573,5,652.3,2...
c_top_200.csv:postgres,https://github.com/postgres/postgres,C,124,0,50,5,41.1,52,1,...
sqlite sorry since it is not hosted on github, and we do plan to add non-github repos in future.
That algorithm seems unnecessarily complicated and includes somewhat dubious metrics when, in my mind, the only thing that really "counts" when it comes to "criticality" are "how many other things use/depend on me", and there are much easier ways to determine this:
1. For languages with a common package repo, how many other packages depend on me? With NPM, for example, it's pretty easy to figure out how many other packages depend on a given package.
2. For "top level" projects, look at downloads.
My guess is just looking at either (or both) of those metrics would give you better results.
I seriously hope no actual decisions about resource allocation etc are made based on this.
A few of your top-10 I can agree with, but when you’re saying a home-automation package (“core”) is more critical than something like pytz then something has gone terribly wrong.
Did i forget military, e.g. weapons?
One exception is inductive automation ignition HMI which uses java
A lot of critical infrastructure software is not active and not often spoken about!
Case in point: The https://www.cip-project.org/faq project highlight the needs of very-long-term support for OS components that run on critical infrastructure such as power stations.
The https://www.cip-project.org/faq project is based on Debian. Very little of it is on github.
by counting the numbers it becomes pretty blindingly obvious what the critical dependencies are. as mentioned in another post above, bash and glibc6 are blindingly-obviously high on the list... yet the GNU Project receives an unbelievably low amount of funding despite their critical importance.
likewise, this particular bug in binutils ld, which centres around the incredibly short-sighted "4GB should be enough for anyone" removal of Dr Stallman's memory-resident algorithms in the late 90s, is having some very serious consequences:
https://sourceware.org/bugzilla/show_bug.cgi?id=22831
yet because there's no money not even from redhat nobody's looking at it.
likewise: PAM no longer has a proper maintainer, and hasn't had for... a decade?
these are projects that people are relying on yet completely forgetting they're a critical part of the infrastructure!
why? because, just as rhencke said above: they're not on github, they've not got "unnecessary changes" which are counted as "activity to be glorified and worshipped".
abharya: i heard on slashdot the intent to start from github, to exclusively focus on github. this will turn out to be a serious mistake.
* Critical projects may have very little activity/maintenance. For example, Bash 4.0 to Bash 5.0 was only 123 commits over 8 years. But, Bash is a absolutely a critical project (ask any org about how much work they had to do when affected by https://en.wikipedia.org/wiki/Shellshock_(software_bug) ).
* A measure of criticality should understand _as many of the various forms of dependence on software_ that may occur that it can. Dependencies can take many forms, such as:
a package manager resolving a dependency
a user purchasing a mobile phone with software pre-installed
a user visiting a website (react/jquery/etc)
etc.
* Criticality should understand if, how, and when dependencies are updated. For example, fixing a bug in Chrome and distributing that fix to 80% of users in 1 week is feasible. Fixing a bug in Bash and distributing that fix to 80% of users in 1 week is not so feasible.
unfortunately, as legal advice, those google employees (right the way to management) do not have the backbone to say, "err no actually, GPL-licensed code is the critically strategically important leveller that forces aberrant companies to collaborate rather than sponge off of underfunded projects".
i mentioned in another post: github "glorifies" the person and the changes that they make. "look at mee! look at mee! i'm making a commit! i'm wiping my backside now! aren't i great!" which gets you precisely zip in terms of actual strategic value.
changes measure change.
people will tell you - if you let them - by providing you with the information needed to make a qualitative assessment.
so.
provide a type of wiki/website that allows qualitative assessments to be made, on a per-library / per-project basis. then put the metrics (the "criticality value") onto that.
pre-seed that wiki/website with stuff from github if you feel so inclined but DO NOT limit the wiki/website to exclusively github. i repeat again: doing so would be a disastrous mistake.
With one of my open source projects I've got to extreme lengths to test it under a huge number of compiler configurations, so it compiles on anything with no warnings. I get nearly zero bugs filed when people port it to esoteric platforms.
When randomly searching a while back, I found a blog post someone wrote about getting my project running on Arduino [1], something I had never tried. Turns out they just had to set some configuration flags and everything worked fine. They therefore filed zero bugs, and it resulted in zero activity for my project. They didn't even tell me about it!
I was extremely pleased that it worked of course, but disappointed that my project continues to look like it's dead. I imagine Google's criticality score for my project is near zero.
[1]: https://www.thingforward.io/techblog/2017-08-03-compiling-lu...
I wrote a script that lists the number of dependents for each package in Guix by traversing the package graphs:
https://gist.github.com/mbakke/f354272666fbef09c5229f7b85377...
Running it takes about 16 seconds on my laptop, and piping to 'grep -v bootstrap | sort -rn | tail -n 20' gives:
(GCC, glibc and binutils are missing for complicated reasons, but should be up there with Guile)By changing (all-packages) on line 26 to:
We get the most popular Python projects: This approach misses "end user" packages such as browsers and QEMU, but provides some insight into high value targets.For end user packages, Debians "popularity contest" can be useful:
https://popcon.debian.org/
source-only distros include macports, gentoo, and so on.
The graphs I'm traversing are the build-time dependency graph of all packages. I could also traverse run-time dependency graphs, but then I'd need to actually build all the things, because those are not known up-front unlike Gentoo and Macports.
The reason the toolchain is missing is because it is an "implicit input" as opposed to a "normal" input, so it's invisible/transparent to the query I'm making.
So a highly dependent project that has plenty of support for its needs should not need to be brought to anyone's attention. I'm guessing python, llvm, clang, chromium, webkit, C#, visual studio code, off the top of my head are all well funded and supported open source projects.
I don't know what good examples of under supported but critical open source projects are. I guess I've read that OpenSSL was massively under supported but apparently the solution chosen was to throw it under the bus and promote libssl or boringssl or something like that (completely out of my expertise)
Isn't that a pretty significant difficulty for the interpretation of this number, though? Couldn't there be some package that is installed by default in, say, Ubuntu (and that Ubuntu can't boot without), so therefore millions of users are using it -- but perhaps the "downloads" seen by GitHub or SourceForge are mostly by OS packagers or by developers contributing to that package. Whereas if there is something where, for some reason, end-users customarily or commonly get it from GitHub or Sourceforge, it might look more widely used even though it is actually less used overall, potentially even by orders of magnitude.
(Like GNU coreutils, at an extreme, is probably not that popular at all for end users to download from a source control system, yet it might be running in tens or hundreds of millions of devices, which could not even boot without it.)
There will always be edge cases and scenarios we are not taking into account, please provide feedback on issue tracker and provide any suggestions so we can account these.
This ranking is very flawed if I can literally go to Best Buy and get a replacement for "critical" software.
It appears 1.0 is most-critical, if you dig into ranking lists.
elasticsearch at the top? flink? It is first time I hear about it and I work in Java shops for quite some time.
But yeah, the list is bizarre. It is more of "list of cool things to read about that you dont really need" then "critical list".
For example, it counts the number of issues as determining criticality. If there is any kind of money attached to this score (now or in the future), obviously this is going to encourage people to introduce more bugs into their projects.
https://opensource.googleblog.com/2020/12/finding-critical-o...
Critical to me means “what’s the thing I have to pay attention to or I’ll suffer consequences”. Having a dependency on a non-popular crate is definitely an increasing score of criticality for that dependency. I may have misread but the fatal error in the metric to me is that popularity of a project increases its criticality when it should decrease. A popular project with lots of contributors means it has lots of stake holders already and a way to successfully manage that. What you want is the small independent projects that popular projects depend on. In other words, what’s the smallest, hardest to notice malicious change I can make in the supply chain of software development to enact the most disruptive change?
> What’s the smallest, hardest to notice malicious change I can make in the supply chain of software development to enact the most disruptive change?
We are working on this problem as well. Trying to identify dependency trees and which smaller projects are the most widely used (indirect deps) and will impact the most on the critical projects in the list.
If you’re just proving out the tech then do that but the current publishing of the most “critical” projects is laughably wrong (as I’m sure you’ve heard from the comments on the page). The project isn’t a dead end but any attempt to use the data right now to drive decisions is a lost cause. Maybe you have some secret sauce that makes it work better for internal Google projects but if that’s the case that methodology should be published as well so that people can get an understanding of how Google uses this more effectively.
I’m just contrasting this to a paper Google published recently about how to measure uptime more effectively in cloud. Every step of that paper was eminently approachable and didn’t have any obvious philosophical flaws that stood out to me (we can argue about them but any potential arguments were called out).
Again, I applaud the attempt. This is a super important problem and trying to tackle it is laudable. Getting the right metrics is crucial and this is being released far too early with a presentation that makes it seem far more complete than it is.
what i strongly recommend that google do - instead of doing this work which is, as you've probably noticed from the comments, well-meaning but highly likely to be biased - is:
don't tell them what to do with the money: leave that up to them. NLnet is extremely good at ensuring that money is used effectively, by requiring that they come up with a project plan involving milestones. they do not just "dump money at the developer", which has a known high historically-backed probability of causing more harm than good: they require the milestones to be completed, 100%, before the money is paid.note, here: they do not use algorithms to assess a project: they use people. they also assess the proposal against the usefulness of achieving key objectives.