Google Plans to Calculate 'Criticality' Scores for Open Source Projects (thenewstack.io) 40
Programming columnist Mike Melanson writes:
As part of its involvement in the recently announced Open Source Security Foundation (OpenSSF), Google has penned a blog post outlining one of the first steps it will take as part of this group, with an attempt at finding critical open source projects.
"Open source software (OSS) has long suffered from a 'tragedy of the commons' problem," they write. "Most organizations, large and small, make use of open source software every day to build modern products, but many OSS projects are struggling for the time, resources and attention they need."
So as a way to address this problem, and help fund those projects that need funding, Google is releasing the Criticality Score project. The project gives projects a criticality score (a number between 0 and 1) that is "is derived from various project usage metrics" such as "a project's age, number of individual contributors and organizations involved, user involvement (in terms of new issue requests and updates), and a rough estimate of its dependencies using commit mentions." From there, you can also add your own metrics, if you see fit...
Abhishek Arya, one of the project's creators, points out that the project is still in its initial phases and welcoming feedback on "any ideas on metrics we can use." Arya also notes that the project is currently limited to ranking open source projects hosted on GitHub, but "will be expanding to our source control system in the near future."
"Though we have made some progress on this problem, we have not solved it and are eager for the community's help in refining these metrics to identify critical open source projects," the blog post announcing the project concludes.
"Open source software (OSS) has long suffered from a 'tragedy of the commons' problem," they write. "Most organizations, large and small, make use of open source software every day to build modern products, but many OSS projects are struggling for the time, resources and attention they need."
So as a way to address this problem, and help fund those projects that need funding, Google is releasing the Criticality Score project. The project gives projects a criticality score (a number between 0 and 1) that is "is derived from various project usage metrics" such as "a project's age, number of individual contributors and organizations involved, user involvement (in terms of new issue requests and updates), and a rough estimate of its dependencies using commit mentions." From there, you can also add your own metrics, if you see fit...
Abhishek Arya, one of the project's creators, points out that the project is still in its initial phases and welcoming feedback on "any ideas on metrics we can use." Arya also notes that the project is currently limited to ranking open source projects hosted on GitHub, but "will be expanding to our source control system in the near future."
"Though we have made some progress on this problem, we have not solved it and are eager for the community's help in refining these metrics to identify critical open source projects," the blog post announcing the project concludes.
Obligatory xkcd (Score:5, Informative)
Critical = someone wants to pay for it (Score:4, Insightful)
Mod parent up for Funny. However I want to comment on a different aspect. Ergo the change of Subject this time.
I think "critical" is a poor way to think about it. What matters is whether or not the costs are covered, including the costs of the programmers' time in creating and maintaining the software, but also including any ongoing costs the software might incur and even preventive (insurance?) costs as in the case of damages if the software is not secured against new exploits. There is too much confusion about economic freedom versus freedom of choice versus not paying cash on the barrel head. Amplified by the confusion of the negative definition used since so many NPOs are involved in OSS and the basic idea of an NPO is a negative definition against the positive idea of profit. Cost recovery makes sense, but lack of profit doesn't really work, because you can't sustain permanent losses and you can't juggle zeros forever.
In terms of solutions (my favorite perversion as Slashdot 2020 sees things), I suggest that OSS should be paid for in advance. That's not to guarantee that every OSS project will succeed or even be completed, but that the costs should be assessed in advance and the money should be flowing continuously as the project is worked on (and maintained). As much as possible, each large project should be broken into small increments with clear success criteria. If enough donors step forward to cover the costs, then the money will flow into the project. If not, then the wannabe developers should rethink their proposal and improve it.
My proposed mechanism for doing this would involve a CSB (Charity Share Brokerage) to handle the money and monitor the projects. I've written about this topic a number of times, but the elevator summary is that the CSB will earn its operating revenue by making sure project proposals are complete and plausible, by helping to publicize the projects to prospective donors (perhaps working with journalists and websites to do so), by evaluating the completed projects against their success criteria, and by reporting those results to the donors and the public. Lots of details in my earlier comments, but feel free to ask questions. At this point this CSB part seems (to me) to have become intuitively obvious to the most casual observer. (You don't have to ask politely, but I'm more likely to answer politely if you do. (It seems like some folks on Slashdot 2020 regard politeness as worse than perversion.)) Even better if you have a better solution approach. (TL;DR is NOT a solution approach, though it's another new tradition of Slashdot 2020.)
When you think about things from the perspective of recovering costs over time, there are implications. One that I haven't written about here is the security of software that accesses the Internet. Any program that does that needs to be perfect (which never happens) or it needs to be maintained to be secure. Not sure what to call this solution approach, though I've been thinking about it for a while and various companies (especially Apple and some gaming companies) are implementing parts of it. Essentially such software should phone home every time it is invoked to see if it's still safe. Basically the software should checksum itself, perhaps using a seed sent from the checker, to confirm that it's uncorrupted and be warned if there are any new versions. If there's a new version, then the user should upgrade or the software will refuse to run or only run in a limited safe mode, perhaps blocked from network access. The check should also include checking for ongoing costs associated with the software. (In conjunction with the CSB I've described that part is "ongoing-cost projects" paid for in advance, perhaps annually or quarterly.) The same check could include recommendations for newer and better software, though my basic position remains that the software deserves to keep on living as long as there are enough people who want to cover the costs.
What the heck. The check could also include an option to include a donor's n
Re: Critical = someone wants to pay for it (Score:2)
Sooo GitHub sponsers?
Re: (Score:2)
If so, then great. (The "so" = "GitHub sponsors perform all the functions of the CSB".)
On your information I duly found https://github.com/sponsors [github.com] which seems pretty good, though the examples they give seem subject to improvement. Also, I think the tax deduction thing is mostly a red herring, especially for small wannabe donors like me. But I feel like I should apologize for not paying much attention to GitHub recently. When did this GitHub Sponsors program start?
I'm trying to map this to the CSB idea, but
Re:Critical = someone wants to pay for it (Score:4, Interesting)
In terms of solutions (my favorite perversion as Slashdot 2020 sees things), I suggest that OSS should be paid for in advance. That's not to guarantee that every OSS project will succeed or even be completed, but that the costs should be assessed in advance and the money should be flowing continuously as the project is worked on (and maintained). As much as possible, each large project should be broken into small increments with clear success criteria.
you may be fascinated to know that this is precisely and exactly what http://nlnet.nl/ [nlnet.nl] do. the biggest concern with google's approach is that they'll end up with completely wrong metrics. the ycombinator comments are already stock-full of warnings that they're using "changes" and "popularity" as "criticality metrics", where in fact the "number of changes" is a pretty good measure of the *exact opposite*.
NLnet use *people* to assess the importance of the project, not some arbitrary statistics.
Re: (Score:2)
Thank you very much for that lead. Looking at the website now... Will try to form an assessment soon. Hopefully even be motivated to send some money.
(If it does do what I hope, then perhaps there is even a mechanism to suggest new projects... No satisfying me, eh?)
Re: (Score:2)
When you think about things from the perspective of recovering costs over time, there are implications. One that I haven't written about here is the security of software that accesses the Internet. Any program that does that needs to be perfect (which never happens) or it needs to be maintained to be secure. Not sure what to call this solution approach, though I've been thinking about it for a while and various companies (especially Apple and some gaming companies) are implementing parts of it. Essentially such software should phone home every time it is invoked to see if it's still safe. Basically the software should checksum itself, perhaps using a seed sent from the checker, to confirm that it's uncorrupted and be warned if there are any new versions. If there's a new version, then the user should upgrade or the software will refuse to run or only run in a limited safe mode, perhaps blocked from network access. The check should also include checking for ongoing costs associated with the software. ...
I suggest you think about this some more. :) Consider something like OpenSSL, very vital and accesses the network. Yet run that inside an enclosed DMZ where it can't hit the outside world for your checks (it would be used for devices inside to talk to each other with ssh and scp). The library and programs that depend on it too are always upgraded only because a new application system is released, not because some new version of a library/program is out. That's to ensure the larger system works the same way
Re: (Score:2)
Not really misunderstanding what I wrote so much as looking for worst-case counterexamples? I think the main branch in thinking is that you are talking about the kinds of software where there is a large kernel of criticality and there are paying customers who are willing or even required to pay in advance keep the software up to date, and not just the critical parts.
I am thinking about more general applications that already check for their own upgrades, but especially apps and browser-resident systems. As i
Re: (Score:2)
Re: Obligatory xkcd (Score:2)
And add Apache Httpd to it.
Some people may claim PHP, Python and GCC as well.
Linux is also a pretty important component. BSD as well. But there are a few items that seems to be forgotten too, like kernel drivers in Linux for some aged hardware.
Tragedy of the commons? (Score:4, Insightful)
Mozilla Firefox (Score:2)
Re: Mozilla Firefox (Score:2)
Of course. I am sure what Google means is critical for them.
NBD as Mozilla is stinking rich. (Score:2)
Mozilla is so wealthy it can choose missions unrelated to Firefox, which I suspect has become boring to those running the show.
Chrome devoured Firefox market share but that doesn't seem to concern Mozilla nearly as much as its "social mission" thanks to the ongoing moral panic consuming the FOSS movement because it's more exciting than code. When people run out of interesting things to do they pursue hobbies instead. They cannot be directed to work on X project if they don't care to, and if leadership don't
CoC (Score:5, Insightful)
Googlers work on worthless apps all day (Score:1)
Citation index (Score:2)
Academic research has long been scoring impact of work based on "citations". The more your findings are used by other scientist, the more important it is. Yes there are different formulas, like H-Index and so on. And you need to discount self-references, and collaboration from group members. However the basic idea is sound.
It is very strange we did not have such a metric for OSS so far. We already have most of the data in DEB and RPM databases, and the rest can be inferred from similar repositories.
Gentoo (Score:3)
Gentoo based systems already have basic tools to find chains of dependencies, as everything is built from source.
That's also Google PageRank, which killed Excite, (Score:2)
That's also essentially the same thing as Google PageRank, the algorithm that caused Google to beat all of the established search engines like Lycos, Excite, Hotbot, Yahoo, etc.
What does Google think of Rocky Linux? (Score:2)
Re: (Score:1)
I certainly hope not. Considering the fates of other projects and services that Google took under its wings -- and then killed -- I hope they stay as far away as possible.
Funding? I don't see any funding mentioned (Score:5, Interesting)
"So as a way to address this problem, and help fund those projects that need funding, Google is releasing the Criticality Score project."
The Google project doesn't seem to mention funding at all. The Linux Foundation project doesn't appear to be offering any funding either. Basically they both seem to be trying to tell other individuals and groups where they should spend their time and money.
I have my doubts either one is likely to gain any traction. I don't particularly see the point, either - are there really lots of people looking around, saying "Google, Linux Foundation, please tell us what to do!"?
Re: (Score:3)
Google does provide finding, will provide more (Score:2)
This isn't 20-years-ago Microsoft, this is Google.
They do fund open source projects. Firefox being one of the most discussed since it's well known to consumers and totally dependent on Google funding.
If Google were going to quadruple their funding of critical open-source projects, they'd first want to identify the critical open source projects. Then they would determine the needed and appropriate levels of funding for each. So this story covers a necessary and proper step in increasing their funding of ope
Ps - 9% of Google employees write open source (Score:2)
Also, beyond just mailing out money, about 9% of the people Google employs committed code or docs to open source projects last year. So they are also directly paying the salaries of open-source contributors.
I think a lot of the public perception of open source discounts how many of the contributions come from full-time developers doing work adding features that will be helpful to their company or fixing bugs that affect their company. Most of my contributions fall into that category. For example, when I fix
Re: (Score:2)
I think a lot of the public perception of open source discounts how many of the contributions come from full-time developers doing work adding features that will be helpful to their company or fixing bugs that affect their company.
I know there are people who still think that way, but I don't know if it's "a lot" anymore. Anyone who's ever read the changelog for the Linux kernel, or that of most other widely-used FOSS projects for that matter, knows the vast majority of commits come from professional developers - especially Red Hat employees.
Re: (Score:2)
Firefox isn't exactly a little-known open-source project. Also, as I recall, Google is getting their search set as the default in Firefox for that money. They pay Apple billions a year for similar reasons.
Regarding your second paragraph... did I miss something? (that's a serious question) Was there a statement somewhere saying that Google is increasing the money they're providing to non-Google projects - and that's why they're doing this?
Re: (Score:2)
> Was there a statement somewhere saying that Google is increasing the money they're providing to non-Google projects - and that's why they're doing this?
I was responding to the statement that Google didn't announce more funding for these projects before they started talking about how they want to identify critical OSS - so that means they are just going around telling other people how to donate their money. I was pointing out that they need to identify the projects BEFORE they fund them. It's quite prem
Introducing money is a recipe for disaster (Score:1)
We don't do it for the donations (Score:2)
Anything that changes the relative distribution of donations could change the behavior of people who are doing it to get donations. I'm going to say that's well under 1% of open source developers.
Most of us fix a bug because the bug is causing a problem for us, not because we're trying to get a donation. We add a feature because we want or need that feature, not because we think it will get a donation.
Re: (Score:1)
Re: (Score:2)
Based on my 20 years in open source, I don't think so.
When our servers are locking up because of a kernel bug, costing me a significantly amount of money, I really, really don't care if there is a $20 donation or not. I'm going to fix the bug that bugging us. When I worked for the Texas A&M University System, 50% of my job was writing open source code to extend an OSS "online campus" (LMS) system called Moodle. If we needed to have a certain type of question on the exams or quizzes for the 50,000 stud
Re: (Score:1)
Dependency graph (Score:2)
A lot of languages have package repositories, one good metric would be a dependency score: the fraction of packages in the repository that, directly or indirectly, depend on this package. The greater that number, the bigger the impact of anything that affects the package.
How many factors have the wrong sign? (Score:2)
What is the score of some of the very old FORTRAN math libraries that are critical for science but aren't touched because they aren't broken and are feature complete?
I think many of the factors are oversimplified. The number of issues has to be tired to the size of the project. The numbers of comments has to be related to the quality of those comments as well.
There are some things here that are good as a low criticality score could mean a project is so small that its use is risky but a high criticality s