The "malicious" code at the end of the advisory looks like nothing more than a beacon announcing it was installed?
edit:
get current working directory
get username
get hostname
concatenate the last 3 together
obfuscate(/encrypt?) this string
send the result as a http request to 121.42.217.44 (the value of the base64 string)
This is correct. In general, though, most packages don't rely on urllib3 directly, but on `requests`, which uses urllib3 but provides a friendlier API and built-in SSL cert verification.
It's not generally true that built-in packages which also appear on PyPI are malicious.
Many batteries-included packages are also maintained outside of CPython. This is because: (1) in many cases they existed outside prior to being included in CPython, (2) they can experiment with new features before they're included in the CPython version of their package.
Not for me. There is space at the end between the closing bracket and the apostrophe. Maybe you did remove this space when you corrected the smart apostrophes.
Yes, but they do filter bash_history client side, only transmitting pip-related commands. They did this to find additional common typos. The relevant code:
def get_command_history():
if os.name == 'nt':
# handle windows
# http://serverfault.com/questions/95404/
#is-there-a-global-persistent-cmd-history
# apparently, there is no history in windows :(
return ''
elif os.name == 'posix':
# handle linux and mac
cmd = 'cat {}/.bash_history | grep -E "pip[23]? install"'
return os.popen(cmd.format(os.path.expanduser('~'))).read()
This is interesting in conjunction with the recent post about Python's popularity because that may be a weakness exploited here [1.]. It's easy to use and install and get libraries for anything, and apparently libraries for infecting your machine :(.
The problem is that the vetting process of PyPI is completely inexistant. This has happened many times in the past, the last time I remember they uploaded a few libraries called "bs4" and stuff like that.
Package managers seem to be an increasingly popular attack vector. It's only luck that none of the attacks have been particularly malicious yet. Considering how many package manager downloads go to a server in a datacenter, a widely distributed malicious package could control a botnet with extremely high throughput, or wreak havoc on any databases it comes into contact with.
It's only a matter of time before something like this happens. A big part of the problem is that application package managers, like pip or npm, are far less sophisticated than those of operating systems, like aptitude or yum. It needs to be easy for developers to open source their code, and to mark dependencies with precise commit hashes, but the download also needs to be secure and verifiable. There are many difficult tradeoffs to consider in terms of usability, centralization, security and trust.
More needs to be done by package managers to warn end users.
One scenario that worries me is where apps age and use popular trusted dependancies (e.g. gems on Github).
When those gems stop being maintained but need to be updated to work (say with latest OSX) - it's common to quickly look at the latest forks available and select the one that now works correctly - but without a detailed inspection of the new code it's potentially kryptonite for a production datacenter.
Maybe most of those 500,000 packages simply shouldn't be trusted.
There's a precedent for curated subsets of package ecosystems. Stackage for Haskell is an example, although it doesn't have security as the primary goal.
I don't think we should focus on actual audits of packages. Just checking that packages seem basically credible seems like a better approach because it's doable.
Credibility is an easier check but still tricky. Many of the package are uploaded by anonymous or pseudonymous authors, so there's no easy way to even tie that to an IRL identity, let alone check for credibility.
I'd agree that a curated small package repository would be a better way to address the problem, but the market doesn't seem very interested in that as a solution.
I think it might just not have happened yet. The npm community can be pretty creative and enthusiastic!
I don't think IRL identities are necessary for what I imagine. It's more like establishing a basic set of packages that have been around, have communities of committers, reverse dependencies, etc.
Maybe we would even make a starting assumption that the transitive closure of dependencies originating with a set of high profile packages are "approved".
I'm thinking aloud but I think there could be a reasonably pragmatic way to get this started...
and yet, I don't see substantial movements relating to package security and trustability in these repo's. To be clear I'm not suggesting these two are any worse than others, they're just large repo's who have had incidents in the past.
The problem here (to my view) is that increasing the security of package repo's will slow down releases (additional checks take time) and cost money (additional security, hosting etc) and until there's a market demand for those service, they won't happen.
I'm skeptical too. But if I think like a sci-fi writer I can vaguely imagine ways for it to actually happen. That open source maintenance happens at all is pretty remarkable, so I think this thing, with an appropriate concept and some good tools (with emojis in their command line output), is at least vaguely plausible...
I don't see any reason it can't be done, either, in theory, and that's with the manual approach. Fancier ideas are viable too, but 500k is still relatively tiny and a manually tractable number. Incremental reviews starting today by many coordinating groups in the node community would take a while to complete, maybe a few years, but with some sensible ordering heuristics like e.g. the most downloaded first, or the most suspicious names first, some value could be produced quickly. But it won't happen, package vetting isn't really a value in these communities. (And that might not really be a bad thing, at least for now...)
Five hundred thousand packages, am I reading it right? I dont believe top ten OS package managers combined would reach that number. Either this is a typo or it's crazy.
There's nothing especially awful about left-pad being its own package, the disaster was because a huge number of developers were betting on npm to somehow be highly available (despite being donated by its admins at no cost and with no committed SLA) rather than vendoring their deps.
One can both vendor and use the package manager to fetch updates. Just add the node-modules directory to your VCS.
The thing with node is that AFAIK it requires you to have libraries for what in most languages would be in the standard library. Maybe someone should start a "stdnode" project where the most popular / successful libraries for generic tasks are integrated into a dependable, maintained de-facto standard library, with an eye on quality and sanity, and community / Joyent funding.
Those service already exit, e.g. https://www.sourceclear.com/ whilst I hope they're doing well, I don't think they've made significant in-roads into the volume of people using open source software library repo's.
Yes, Conda [0] is a package manager designed by Continuum Analytics (now Anaconda, Inc.) to support their Anaconda Distribution [1]. The distribution is free, and the Conda client is open source. However, Anaconda sells several enterprise products, including an on-premise Conda server ("Anaconda Repository").
In general, Conda does more package verification than pip, and the packages in the Anaconda distribution are more thoroughly vetted than PyPi. Conda-Forge [2] provides an escape hatch for less-vetted community code.
It always worries me when I install a well-known or large package from npm and it ends up downloading dozens of dependencies maintained by disparate and unaccountable github users.
I'm always fascinated by the amount of trust being exhibited by the developers of some node projects I've seen. Their projects have an order of magnitude more dependencies than I'm used to - and at the other end of each one is someone publishing some small module to npm with an unknown amount of review. I feel safe(r) installing dependencies from apt because I know the processes the Debian community follows before packages are included in the official repos.
Another fun fact to consider is that with many package formats, you can execute arbitrary code at install time so if a malicious package can get into a repository, it's very likely to start compromising systems quickly.
Whilst a package manager repo. compromise would be the biggest bang in terms of attack, compromising the credentials of the developers of popualar libraries would be an easier attack (and indeed is already happening https://twitter.com/chrispederick/status/892768218162487300)
I almost never see this. Even on systems that are only running a single python project, I only ever see folks use virtualenv. The only time I ever see things installed with sudo is when the package is being installed in a docker container.
With Python you can execute arbitrary code at import time... you don't even have to get to the install process. I've seen packages on Pypi that try to "sudo apt-get install ..." when the setup.py file is imported.
Also, even without sudo there's absolutely nothing stopping you (for example) downloading a cryptocurrency miner, or DDOS tool, or something, and starting it up to run in the background.
I was recently very surprised to see an expensive security scanner which builds apps as the service account apparently without sandboxing. A package manager which executes code or which is exploitable would give you access to what is very likely an interesting account & data, and in at least some cases might not leave many clues behind. PyPI, etc. at least has centralization and immutable versions but who knows what's serving some random repo, tarball, etc. which might not even be a direct dependency?
This is why we are working on integrating TUF (The Update Framework) signing into the OCaml OPAM package manager. See https://github.com/hannesm/conex-paper/blob/master/paper.pdf for the talk from last year. There's one more iteration required on the implementation before we're happy with it, but we are aiming to get this live on the OPAMv2 package repository some time in 2018 for all the publicly available OCaml packages.
OPAMv2 also exposes sufficient hooks during the build process for using OS sandboxing during builds, and disconnecting network access/etc. It would be nice to factor this out to be more OS independent (e.g. for all the `unshare` tricks on Linux, or the sexp-format for sandboxing on OSX) in the future.
This to me is the nightmare scenario. Well one of the two, the other one being that a developer of an obscure library I use has their password to PyPI compromised and a bad actor uploads a backdoored version of the library.
Fundamentally, the reason this is different from how thinks like Linux distos work is because Linux distros have maintainers who are in charge of making sure every new update to one of their packages is legit. I am sure you can try to sneak malicious code in, but it isn't going to be easy.
I am not advocating that PyPI (and npm) adopt the same model. That would be too restrictive. But maybe just showing the number of downloads isn't the best way to assure whether the package is legit. Perhaps some kind of built in review system would be nice.
A review system unfortunately isn't likely to be practicable with current development models. npm alone has over 500,000 packages (http://www.modulecounts.com/) so even a one time review isn't going to happen.
If people want a more trusted solution the likely outcome is that they'll need to use a smaller more static set of libraries and then either do the audits themselves, or outsource that to a 3rd party.
Ofc with current speeds of change and deployments, it doesn't seem likely that many companies will adopt that model.
> npm alone has over 500,000 packages (http://www.modulecounts.com/) so even a one time review isn't going to happen.
But at least the modules with the most downloads (webpack, react, or stuff like left-pad) could be vetted, and especially npm could implement a 2-or-more person model - basically, everyone with publish access can upload a new artifact, but to actually have it distributed to endusers, a second person would be required to sign off.
Full code review while not bad is probably not going to stop attackers. Often I can barely understand what the dev intended with non hostile code. Something more along the lines of strace on the install and run down any connects or execs. Runtime is different.
That's the thing. I worry less about popular packages. I can check that Django's GitHub repo links to PyPI and vice versa. But a random package to parse DSN's? I don't know it from Adam. I want to use it, and lots of others do too, but not everyone is going to review it. Maybe just a button on the package that says "I found insecure code!" Would be good.
> That's the thing. I worry less about popular packages. I can check that Django's GitHub repo links to PyPI and vice versa.
I worry about the most popular, and there the small and next-to-unmaintained. Just think back to the left-pad desaster that broke builds all over the world and imagine it was not a deleted package but an update containing malware. I assume there are lots of such "hidden gems" where the maintainer has gone away... the consequences of hacking just one improperly secured account are severe.
Indeed they do, and you can get some libs for Node/ruby etc there, however most companies, from what I've seen choose the option of using direct access to npm/rubygems etc.
> Fundamentally, the reason this is different from how thinks like Linux distos work is because Linux distros have maintainers who are in charge of making sure every new update to one of their packages is legit.
Because the person who pushes the code to the public repo is not the same person who makes sure it isn't malicious. You have a review process. Nothin is stopping me right now from creating a PyPI package called Django2.0 and having some poor souls download it. Or creating a tiny but useful utility, having it become popular, then introducing an update with a backdoor.
I think a more Linux-like approach to package repos is better - a curated package repository run by volunteers in maintainership roles. Then you have a human being verifying the upstream and keeping malware out, and get more consistency across packages as a bonus. If you want your package added it's as simple as sending an email and provides a new avenue for people to contribute to the success of the ecosystem as package maintainers.
When you make the next big thing, consider this approach.
Maybe you're right, but I see one possible downside that is quite important.
I have encountered the case "the package has an important bugfix but is not yet published on PyPI" way more than once or twice.
With the intermediate maintainers, that's going to get worse.
I believe namespaces and signatures are the way to go. With a special privileged namespace for the curated widely known packages (e.g. SciPy or Django) - a little like it's on the Docker Hub, where curated mainstream images are just "debian" or "python" but anyone can upload e.g. "jdoe/debian" if they need some customization.
PyPI should also run a build to audit behavior which would be fairly easy to implement. A submitted package would just fail if it access the network or privileged files during compile unless unique needs are called out in an spec file.
I do wish that `--user` was the default for pip.
It is also a pity that trivial Debian bugs like this block adoption of non sudo pip installs weren't ignored.
Although Debian/Ubuntu default to --user on pip people resort to sudo because the current standard user bin directory isn't in the default path due to a regression.
I may start a project to create a apparmor/selinux wrapper for pip to audit and restrict access to sensitive resources. I actually have a fairly heavyweight version in place on my build pipeline to detect new dependencies. I add the files/network resources that a build accesses outside of the testing stage to the build artifacts. But it wouldn't be cross platform enough for Windows/Mac.
1. This would require PyPI to provide computing resources to build the packages (for all OSes, if the package contains native code). And then, malicious packages would just detect the build environment and avoid exhibiting the unwanted behavior. I think I read that it's not unusual for malware to detect being ran in a VM and doesn't do anything suspicious to complicate detection and analysis.
2. I don't think `pip install --user` adds any significant security. A little bit - sure, but not much. A trivial injection into ~/.*shrc or ~/.profile (I don't think anyone would notice the file was changed until it's too late) would result in full system compromise on the next login and sudo invocation. Same goes if you have ~/bin or ~/.local/bin (or anything user-writeable) in $PATH.
And even with non-root access, malicious software can do a lot of undesirable things (e.g. send spam or steal user data).
---
I believe, signature-based trust (with mandatory code signing) is the way to go. On the first `pip install` ever ask "The package not-expect (1.2.3) is signed by The Spanish Inquisition (key: ...) and was audited by The Python Developers (key: ...). Have you verified the keys and do you trust a) this vendor, b) this auditor or c) both?", then it gets recorded in ~/.pip (for this machine) and ./requirements.txt or ./setup.{py,cfg} (for distribution) so future installations don't ask anything.
To get non-interactive mode (for CI or something), on must either pass something like --insecure-skip-signature-checks (so they mean it) or pre-supply all the trusted keys.
(Not ideal, of course - just a quick idea. Surely, it has a lot of rough edges to polish.)
On point 1) yes so they would have to compile, that is not a huge barrier and it would also result in improved quality of modules just as CI/CD does.
On 2) even if you ignore the much larger attack surface due to running ALL installs as the root user, consider the one-shot opportunities by disabling the protections of capabilities(7)
# capsh --print -- -c 'pip list > /dev/null'
Current: =
cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read+ep
And if --user was the norm, it would be trivial to write an apparmor/selinux policy to protect files like ~/.*shrc or ~/.profile
It is just the basic principle of least privilege. Heck if irssi can bother with an apparmor profile, the maintainers of pip or the package should be able to.
I don't think there are any maintainers that verify upstream code, they only manage packages and updates. Which is actually safer to do without maintainers, completely automatically, as it will eliminate a huge attack surface introduced by a maintainer.
Debian Developers are responsible of the package quality. Most package only well-known software that they are familiar with or read the code. Some do more thorough security audits.
Some security-sensitive packages are maintained by teams to share the workload.
They do a lot that could be and sometimes even is automated, but that doesn't help with security, only weakens it. Which is my point. It's better to automate package generation and trust fewer people, mainly the authors, not introduce maintainers into it.
Authors don't generate distro packages because there's too many distros and each distro needs to make changes that have nothing to do with code. Maintainers are necessary and a package signed by the author is a non-starter.
However, source code can be signed and then used to make a package signed by a distro.
Of course you can do this - all packages are basically just wrappers around upstream code. But you still need someone to maintain the wrapper, and they have to check every new code release to see if there's something in the wrapper that has to change. And there are multiple distros. There's no getting away from maintainers with traditional linux distros.
Code package management is different. The author writes their software specifically to conform to the one code package management system. There's no wrapper glue needed, so you don't need a maintainer. Just release your new code and it fits into the system, and other code/tools/etc can just pick it up and use it.
This works if you constantly update all the software you use everywhere, and is pretty much guaranteed to become a nightmare if you don't. CPAN is probably the most mature software package management system in existence and it's still a nightmare if you don't keep a private repo and tightly manage releases, and you absolutely need a maintainer.
To be clear, what I'm suggesting is to generate those wrappers automatically, instead of maintaining them manually. A script can visit a release page daily, parse it and check for updates. If there is a new upstream release, it can generate a wrapper and let the build system do the rest, produce binaries, test them, etc. When things break, the code needs to be fixed, but it's definitely very far from every release. And you don't have to trust the maintainer of that script anymore or even have a separate maintainer, everything could be reviewed on pull requests with only a small group of people having commit rights to the repository.
That's basically how packages are maintained today, they just don't have as much automation and there's a lot less packages as a result. If you made one package for every software update and had to review each one you'd spend a lot more time reviewing.
Trust isn't an issue in reviewed/maintained repos because you have eyeballs on everything. When anyone can just ship an app/library and release it automatically you get these malicious software issues.
This isn't, in any way, a new problem. I did a presentation on this topic for OWASP AppSecEU 2015 (https://www.youtube.com/watch?v=Wn190b4EJWk&list=PLpr-xdpM8w...) and when doing the research for that I encountered cases of repo. attacks and compromise.
IME the problem will continue unless the customers (e.g. companies making use of the libraries hosted) are willing to pay more for a service with higher levels of assurance.
The budget required to implement additional security at scale is quite high, and probably not a good match with a free (at point of use) service.
Interesting if you think that npm/Rubygems/PyPI are leaving a load of money on the table, why do you think they haven't introduced those services so far...
Because their mission isn't to generate income like a traditional business. But if the income went back to the foundations, like Python Foundation, I think that would make sense.
But income can be also used to help finance their main mission. Obviously they seem to operate fine without strong reasons to expand revenue streams, but I feel like they ignore an opportunity to create improvements for just about everyone.
Anaconda gives a healthy amount to open source, either by donations to foundations like NumFOCUS or paying salaries of contributors. Is that what you're looking for?
ISTM we're just talking about running an alternate, more restrictive registry? npm etc. don't have to play any part in that. This service could be offered by anyone: IBM could do it.
If someone here wants to build a business around this, count me in for NPM (high willingness to pay) or PyPi (lower WTP).
Here's an idea: make it similar to Kickstarter, where customers can commit a certain amount of funds towards a specific package. If the package doesn't "tilt" in a certain amount of time money goes back. Otherwise you vet a point release and add it to your repo. you could offer subscriptions to keep packages updated or handle each update as its own project (with presumably lower costs if a recent release has been audited). Handling dependencies is key as an exercise for the reader
One thing to consider if you're going to provide a service like this:
What happens if a vulnerability nevertheless sneaks through?
The whoever did the vetting could conceivably get sued. So then they might want to take out insurance or try to protect themselves from lawsuits in some other way -- all of which is likely to make such a service even more expensive.
It has to be constrained to something reasonable. You can't guarantee the software is safe, but you can guarantee it is published by someone who is who they say they are, similar to EV certificates for domains. You can also refuse to publish packages with intentionally-confusing names.
"you can guarantee it is published by someone who is who they say they are"
Can you? Positively identifying people seems a pretty tricky and easily screwed up business.
ID's can be forged, and a web of trust requires, well, trust.
I guess such a service could say something like "we got this person's ID (and/or address)" or "here's this key's web of trust", and that would probably be a bit better than what we have today (which is virtually nothing), but it would still be a far cry from "guaranteeing it is published by someone who is who they say they are".
EV certs have a complex verification process that can involve sending a physical representative from the company down to the place of business to confirm its presence/existence.
Bitcoin trading platforms have shown that compliance with AML/KYC regulations can be performed virtually by manual verification of a valid government ID, timestamped photo, handwritten note, and other mechanisms.
A company offering this service would go outside of the keyserver and verify the ID independently. It'd be much more of a "notarized packages" paradigm rather than just "published by 1337PyHax0r-88".
It is true that even extensive manual verification processes dependent on government-issued IDs can be faked, but there's a much higher bar involved.
Any method of software distribution which is not rooted in cryptographic author verification against a fine-grained, user-manageable trust store should be put bellow the sanity waterline, 20 years ago.
Part of my dislike for the Node ecosystem in particular and I am sure others have a similar problem, is the dependency trees are super complex.
Because packages tend to be small and many, and each of those has their own dependencies, you can end up with hundreds of packages installed which is simply impractical to manually review.
It is not node, but we do in fact manually review each package we utilize for our given language because it's feasible and worthwhile as the dependency tree is small in this ecosystem. Each and every package is a possible attack vector whether that be intentionally or just because it's poorly written and we can't simply ignore that because it's the done thing and "the community reviews them".
"Success of the attack relies on negligence of the developer"
How about package manager managers accept their enourmous responsabilty? urllib vs urllib2, one is a virus? Sorry but that is not "negligence of the developer"
The least they can do is create an alias system for common libs or disallow some lib names.
Another easy thing to implement would be a popularity check: "This package was only installed nnn times. Did you mean xxx, or do you want to proceed with the installation of yyy by author dev@g00gle.com?"
Managing supply chain is one the basic principles of good engineering. Not properly vetting your sources is negligence. The problem of course is that computers are really good at amplifying work, including mistakes. So small mistake, like a typo, could have catastrophic impact, like injecting malware that can take over the whole system.
I'm all for security but this hit a nerve with me:
"Success of the attack relies on negligence of the developer, or system
administrator, who does not check the name of the package thoroughly."
Package managers need to do more. If they had an enterprise version that you could subscribe to monthly/annually invoice that you would get enterprises onboard, they are concerned about security and will pay. Developers like us will help encourage it. I'd rather not see some third-party "secure" package managers but make them part of PyPi and send funding to the Python foundation. They are seeking donations but that doesn't work well with businesses. Make it a monthly/yearly service.
Yet another attack vector that doesn't exist at all in Linux distributions but invented by language package managers, sadly.
They solved the issue 2 decades ago by heavily vetting packages before accepting them into repositories. Users are allowed to add and use packages from 3rd party repositories.
Maybe solution to this is creating curated repositories based on publicly open ones and using them by default (and requiring opt-in for using other repositories). Conda for Python and Stackage for Haskell seems like relevant solutions.
There's a certain amount of work (and therefore money) required to do this. That incremental difference is small for a well designed application, but - someone must actually vet and curate the contents of the repo. That tends to slow down execution, leading to scenarios where things like docker a year or two ago from the canonical "trusty" repo were hopelessly behind the "real" docker since docker was evolving so quickly and trusty was by design slowing down.
Each commit that went into trusty required a team to submit and a team to approve. That costs money. ;-)
It is a matter of distribution and release policy and not an inherent limitation of the model.
Stable/lts/enterprise distributions have other concerns like preventing regressions and configuration or behavior changes during lifetime of release.
Rolling distributions like Arch and OpenSuse Tumbleweed on the other hand can move a lot faster but still provide basic vetting wrt security and sanity of new/updated packages.
You are cherry-picking one example involving a library that had a plethora of vulnerabilities from upstream. Contrast it with reviewing and maintaining 50k+ packages, managing thousands of CVEs every year, sometimes even writing security patches before upstream.
Also the project pioneered reproducible builds and implemented build hardening for most packages.
None of that, nor the good that they have done, negates the fact that distributions can fuck up too. And regardless of whether OpenSSL had a plethora of bugs from upstream, this one wasn't one of them. It was only in Debian, because of changes the project had made. Just because it's packaged in Debian by Debian maintainers doesn't mean you're immune to these kind of issues. They're arguably less likely but you'd need to do a comprehensive study of all packages in the repo to get to some usable statistic.
So the PyPi issue isn't a bug. It is an attack by hostiles. There are always bugs but this is a person packaging malware probably as a practice to package worse stuff that runs at build time. If package maintainers have time to respond to valgrind reports on their package they have time to check strace on the installer. Edited: longer rant.
> So the PyPi issue isn't a bug. It is an attack by hostiles.
Arguably. The issue with typosquatting on PyPi has been known and demonstrated for a long time, but nothing has been done about it. Considering there are ways of closing this attack vector, even though it would require some serious work, I'd consider this a bug. It's just a bug that's being exploited now.
This is a totally different issue. Distro package management isn't perfect, but you don't have to worry about a random malicious individual squatting on "opensssl" and including compromised code. That's a whole different ball game than a bug inadvertently introduced during a backport.
Sure its nice (and easier) to use the distro's package management system, but it often just isn't up to date enough. You end up using thing that are a while out of date and may have security flaws as a result.
This is incorrect of rolling release distros. Even Ubuntu is fairly quick to update, which is not a rolling release. CentOS on the other hand can be like pulling teeth. I'm going to be glad to stop dealing with CentOS 6.
For popular packages perhaps, but for many more obscure and niche python packages Ubuntu is often a several releases behind, and that is if there even exists a package to begin with
> Yet another attack vector that doesn't exist at all in Linux distributions but invented by language package managers, sadly.
Language package managers solve the problem that we don't have neither an universal package format that works across all programming language requirements and all sort of OSes, nor the time to create an OS specific package for all sort of OSes.
Solving that problem is orthogonal to vetting repositories.
There is nothing wrong with inventing their own solution if they're solving other problems, what is wrong is not learning from previous examples and fucking up creating problems that have been already solved in the process.
But developers also want a way to get software without getting it blessed by Debian and waiting months/years for a distro release. That's why repositories like PyPI exist and are in very widespread use.
Distro repositories are a great example of 'secure for ideal users'. They give you security if you can put up with a small selection of software and older versions. In practice, we end up working around distro repositories by installing stuff with pip, or PPAs, or downloaded from websites.
I don't understand why they even try, Debian stable seem to have an almost arbitrary selection of outdated ruby and python libraries, at this point that hardly seems worth the effort. Sure I get the idea, but it obviously doesn't work in practice.
Their security methodology also seems heavily flawed, to backport security fixes to older versions is neither scalable nor particularly reliable. I sincerely doubt that Debian can provide adequate security to its almost 50,000 packages. If the security community would invest equal amount of resources that they invest in finding flaws in iOS or Chrome, nothing would be left of Debian but a pile of smoking ashes.
> Debian stable seem to have an almost arbitrary selection of outdated ruby and python libraries
Yet Amazon and other big tech companies have a very similar process of packaging open source software for internal use and relying on "outdated" libraries.
> I sincerely doubt that Debian can provide adequate security to its almost 50,000 packages.
There's a security tracker where you can see how quickly packages are assigned CVEs and patched - sometimes even before the upstream patch is ready.
> Yet another attack vector that doesn't exist at all in Linux distributions but invented by language package managers, sadly
Not really, PPAs (and equivalents like copr or obs or aur etc etc) are mostly vulnerable to similar problems. People do want to install upstream software for various reasons, blaming language package managers for the reduced security of that is imho disingenuous.
> Do digitally signed certificates fit into this usage scenario??
No, because either the package author would have to sign them, in which case you have to choose to trust each package author, or the repository would sign them, in which case there would be no improvement for this current issue, since the repo would sign the fake packages as well.
Any way in which blockchain technology can be used? Like, the transaction becomes the act of the author uploading the code and the repo and user verify the transaction in some form?
To be fair you can do a lot with simple heurestics. New package + large number of downloads || new package || new package author = show users a warning message before installing the package.
This is why I am not a huge fan of using package managers. I like to understand the code we put into our platform, and vet it. And not have it change under us automatically, after that, but review the changes manually before accepting it.
I felt a bit curmudgeonly but we have a responsibility at https://qbix.com/platform for all our apps being secure. I wanted to use repos for each package and manually git pull or hg pull them when they changed.
I was finally convinced by our developers to just use package managers with version pinning. Honestly it's really hard to avoid package managers, especially for all the newer functionality such as Payment Requests or Web Push. Luckily there is version pinning.
We want our clients to feel secure that we vetted ALL the code that went into the platform. So our package json (and composer.json) uses version pinning. We'd rather take a bug report and manually fix it than NO bug report and have a SHTF moment.
It's less of an issue, but it still could be an issue.
Deployed Maven artifiacts from Central are to required to be signed with a PGP key and are only supposed to come from approved hosts. I don't know how strictly that is enforced and how hard it is to become a host, but at least there is some kind of process.
Maven Central also doesn’t allow the removal of artifacts after they've been published, and every artifact requires a unique version and name. And the names are namespaced. So you don't have the issues that you see with npm, where someone can pull a package and break everything people are using, and then some third party can come in and publish anything under the exact same name.
Is this model perfectly secure? No, you still have to trust that the artifact was signed by a non-malicious person from a host that was not compromised.
When a dependency changes, all the projects that directly depend on it should get notified immediately and their maintainers should rush to test the new changes, to see if they break anything.
There is no shortcut around this, because if B1, B2, ... Bn depend on A1, the consequences may be different for each Bk.
The only real secure optimization that can be done is realizing that some of the Bk use A1 the exact same limited way and thus make an intermediate A1b that depends on A1 which those Bk's depend on. These "projection" builds may be automated by eg the set of methods called by the B's.
Anyway, this is the way that iOS does it before iOS 11 comes out to users. They release a beta to all developers. And they even fix bugs in the beta before releasing to the public.
Without beta testing periods, you can get laziness and just auto-accepting of whatever cane out.
There is be an "alpha release" feature in git where maintainers might put out the next version to be tested by all who depend on it. THIS FEATURE SHOULD NOTIFY THE MAINTAINERS SUBSCRIBED TO THE REPO. THE BUILD ITSELF SHOULD GET ISSUES AND RATINGS FROM MAINTAINERS AS THEY TEST THE NEW BUILD. And releases should not be too frequent.
This is the way to prevent bad things from happening. But that also means that the deeper the dependency is, the more levels this process could take to propagate to end-users.
253 comments
[ 2.7 ms ] story [ 262 ms ] thread# just toy, no harm :)
`pip list –format=legacy | egrep '^(acqusition|apidev-coop|bzip|crypt|django-server|pwd|setup-tools|telnet|urlib3|urllib) '`
This incorrectly lists `urllib3` or the `cryptography` package for example, which are perfectly valid packages.
[UPDATE]
Read "tobltobs" comment below. I incorrectly removed a trailing space from the regex.
Many batteries-included packages are also maintained outside of CPython. This is because: (1) in many cases they existed outside prior to being included in CPython, (2) they can experiment with new features before they're included in the CPython version of their package.
https://pypkg.com/pypi/xml/f/setup.py
https://gist.github.com/osteele/198b50a2a208e5bc7e5fb8d010cf...
[1.] https://news.ycombinator.com/item?id=15249348
IP: 121.42.217.44 Decimal: 2032851244 Hostname: 121.42.217.44 ASN: 37963 ISP: Hangzhou Alibaba Advertising Co.,Ltd. Organization: Hangzhou Alibaba Advertising Co.,Ltd. Services: None detected Type: Broadband Assignment: Static IP Blacklist: Click to Check Blacklist Status Continent: Asia Country: China cn flag State/Region: Zhejiang City: Hangzhou Latitude: 30.2936 (30° 17′ 36.96″ N) Longitude: 120.1614 (120° 9′ 41.04″ E)
"Hi bro :)
Welcome Here!
Leave Messages via HTTP Log Please :)"
It's only a matter of time before something like this happens. A big part of the problem is that application package managers, like pip or npm, are far less sophisticated than those of operating systems, like aptitude or yum. It needs to be easy for developers to open source their code, and to mark dependencies with precise commit hashes, but the download also needs to be secure and verifiable. There are many difficult tradeoffs to consider in terms of usability, centralization, security and trust.
One scenario that worries me is where apps age and use popular trusted dependancies (e.g. gems on Github).
When those gems stop being maintained but need to be updated to work (say with latest OSX) - it's common to quickly look at the latest forks available and select the one that now works correctly - but without a detailed inspection of the new code it's potentially kryptonite for a production datacenter.
The problem is one of scale. npm has over 500,000 packages, so no manual review will address their scale over the whole repository.
Until the developer market shows that they'll pay for a more secure service (e.g. package signed, reviews done etc) I doubt much will change.
There's a precedent for curated subsets of package ecosystems. Stackage for Haskell is an example, although it doesn't have security as the primary goal.
I don't think we should focus on actual audits of packages. Just checking that packages seem basically credible seems like a better approach because it's doable.
I'd agree that a curated small package repository would be a better way to address the problem, but the market doesn't seem very interested in that as a solution.
I don't think IRL identities are necessary for what I imagine. It's more like establishing a basic set of packages that have been around, have communities of committers, reverse dependencies, etc.
Maybe we would even make a starting assumption that the transitive closure of dependencies originating with a set of high profile packages are "approved".
I'm thinking aloud but I think there could be a reasonably pragmatic way to get this started...
it's been at least 5 years since npm started getting scrutiny relating to security weaknesses https://blog.andyet.com/2012/03/08/compromising-the-integrit...
and 4 years since Rubygems was compromised http://blog.rubygems.org/2013/01/31/data-verification.html
and yet, I don't see substantial movements relating to package security and trustability in these repo's. To be clear I'm not suggesting these two are any worse than others, they're just large repo's who have had incidents in the past.
The problem here (to my view) is that increasing the security of package repo's will slow down releases (additional checks take time) and cost money (additional security, hosting etc) and until there's a market demand for those service, they won't happen.
[1] https://imgur.com/a/enjvR
Also, they are not vetted and there are much more opportunities for an attacker to sneak in a backdoored lib on the edge of the dependency graph.
Finally, due to vendoring there's no way to receive timely drop-in security fixes for all dependencies from a trusted source.
The thing with node is that AFAIK it requires you to have libraries for what in most languages would be in the standard library. Maybe someone should start a "stdnode" project where the most popular / successful libraries for generic tasks are integrated into a dependable, maintained de-facto standard library, with an eye on quality and sanity, and community / Joyent funding.
In general, Conda does more package verification than pip, and the packages in the Anaconda distribution are more thoroughly vetted than PyPi. Conda-Forge [2] provides an escape hatch for less-vetted community code.
[0] https://conda.io/docs/index.html [1] https://www.anaconda.com/distribution/ [2] https://conda-forge.org/
Whilst a package manager repo. compromise would be the biggest bang in terms of attack, compromising the credentials of the developers of popualar libraries would be an easier attack (and indeed is already happening https://twitter.com/chrispederick/status/892768218162487300)
Running "pip install" as a user that has access only to the virtualenv directory is sounding like a good strategy.
Also, even without sudo there's absolutely nothing stopping you (for example) downloading a cryptocurrency miner, or DDOS tool, or something, and starting it up to run in the background.
OPAMv2 also exposes sufficient hooks during the build process for using OS sandboxing during builds, and disconnecting network access/etc. It would be nice to factor this out to be more OS independent (e.g. for all the `unshare` tricks on Linux, or the sexp-format for sandboxing on OSX) in the future.
Fundamentally, the reason this is different from how thinks like Linux distos work is because Linux distros have maintainers who are in charge of making sure every new update to one of their packages is legit. I am sure you can try to sneak malicious code in, but it isn't going to be easy.
I am not advocating that PyPI (and npm) adopt the same model. That would be too restrictive. But maybe just showing the number of downloads isn't the best way to assure whether the package is legit. Perhaps some kind of built in review system would be nice.
If people want a more trusted solution the likely outcome is that they'll need to use a smaller more static set of libraries and then either do the audits themselves, or outsource that to a 3rd party.
Ofc with current speeds of change and deployments, it doesn't seem likely that many companies will adopt that model.
But at least the modules with the most downloads (webpack, react, or stuff like left-pad) could be vetted, and especially npm could implement a 2-or-more person model - basically, everyone with publish access can upload a new artifact, but to actually have it distributed to endusers, a second person would be required to sign off.
The fact it hasn't been done despite the obvious risks indicates how much demand there is for this feature...
I worry about the most popular, and there the small and next-to-unmaintained. Just think back to the left-pad desaster that broke builds all over the world and imagine it was not a deleted package but an update containing malware. I assume there are lots of such "hidden gems" where the maintainer has gone away... the consequences of hacking just one improperly secured account are severe.
That's what stable Linux distributions do.
How is that different?
When you make the next big thing, consider this approach.
I have encountered the case "the package has an important bugfix but is not yet published on PyPI" way more than once or twice.
With the intermediate maintainers, that's going to get worse.
I believe namespaces and signatures are the way to go. With a special privileged namespace for the curated widely known packages (e.g. SciPy or Django) - a little like it's on the Docker Hub, where curated mainstream images are just "debian" or "python" but anyone can upload e.g. "jdoe/debian" if they need some customization.
I do wish that `--user` was the default for pip.
It is also a pity that trivial Debian bugs like this block adoption of non sudo pip installs weren't ignored.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=839155
Although Debian/Ubuntu default to --user on pip people resort to sudo because the current standard user bin directory isn't in the default path due to a regression.
I may start a project to create a apparmor/selinux wrapper for pip to audit and restrict access to sensitive resources. I actually have a fairly heavyweight version in place on my build pipeline to detect new dependencies. I add the files/network resources that a build accesses outside of the testing stage to the build artifacts. But it wouldn't be cross platform enough for Windows/Mac.
2. I don't think `pip install --user` adds any significant security. A little bit - sure, but not much. A trivial injection into ~/.*shrc or ~/.profile (I don't think anyone would notice the file was changed until it's too late) would result in full system compromise on the next login and sudo invocation. Same goes if you have ~/bin or ~/.local/bin (or anything user-writeable) in $PATH.
And even with non-root access, malicious software can do a lot of undesirable things (e.g. send spam or steal user data).
---
I believe, signature-based trust (with mandatory code signing) is the way to go. On the first `pip install` ever ask "The package not-expect (1.2.3) is signed by The Spanish Inquisition (key: ...) and was audited by The Python Developers (key: ...). Have you verified the keys and do you trust a) this vendor, b) this auditor or c) both?", then it gets recorded in ~/.pip (for this machine) and ./requirements.txt or ./setup.{py,cfg} (for distribution) so future installations don't ask anything.
To get non-interactive mode (for CI or something), on must either pass something like --insecure-skip-signature-checks (so they mean it) or pre-supply all the trusted keys.
(Not ideal, of course - just a quick idea. Surely, it has a lot of rough edges to polish.)
On 2) even if you ignore the much larger attack surface due to running ALL installs as the root user, consider the one-shot opportunities by disabling the protections of capabilities(7) # capsh --print -- -c 'pip list > /dev/null' Current: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read+ep
And if --user was the norm, it would be trivial to write an apparmor/selinux policy to protect files like ~/.*shrc or ~/.profile
It is just the basic principle of least privilege. Heck if irssi can bother with an apparmor profile, the maintainers of pip or the package should be able to.
Some security-sensitive packages are maintained by teams to share the workload.
However, source code can be signed and then used to make a package signed by a distro.
Code package management is different. The author writes their software specifically to conform to the one code package management system. There's no wrapper glue needed, so you don't need a maintainer. Just release your new code and it fits into the system, and other code/tools/etc can just pick it up and use it.
This works if you constantly update all the software you use everywhere, and is pretty much guaranteed to become a nightmare if you don't. CPAN is probably the most mature software package management system in existence and it's still a nightmare if you don't keep a private repo and tightly manage releases, and you absolutely need a maintainer.
Trust isn't an issue in reviewed/maintained repos because you have eyeballs on everything. When anyone can just ship an app/library and release it automatically you get these malicious software issues.
Even that minimal amount of work is enough to prevent such a ridiculous attack like typo squatting.
IME the problem will continue unless the customers (e.g. companies making use of the libraries hosted) are willing to pay more for a service with higher levels of assurance.
The budget required to implement additional security at scale is quite high, and probably not a good match with a free (at point of use) service.
Here's an idea: make it similar to Kickstarter, where customers can commit a certain amount of funds towards a specific package. If the package doesn't "tilt" in a certain amount of time money goes back. Otherwise you vet a point release and add it to your repo. you could offer subscriptions to keep packages updated or handle each update as its own project (with presumably lower costs if a recent release has been audited). Handling dependencies is key as an exercise for the reader
What happens if a vulnerability nevertheless sneaks through?
The whoever did the vetting could conceivably get sued. So then they might want to take out insurance or try to protect themselves from lawsuits in some other way -- all of which is likely to make such a service even more expensive.
Can you? Positively identifying people seems a pretty tricky and easily screwed up business.
ID's can be forged, and a web of trust requires, well, trust.
I guess such a service could say something like "we got this person's ID (and/or address)" or "here's this key's web of trust", and that would probably be a bit better than what we have today (which is virtually nothing), but it would still be a far cry from "guaranteeing it is published by someone who is who they say they are".
Bitcoin trading platforms have shown that compliance with AML/KYC regulations can be performed virtually by manual verification of a valid government ID, timestamped photo, handwritten note, and other mechanisms.
A company offering this service would go outside of the keyserver and verify the ID independently. It'd be much more of a "notarized packages" paradigm rather than just "published by 1337PyHax0r-88".
It is true that even extensive manual verification processes dependent on government-issued IDs can be faked, but there's a much higher bar involved.
https://twitter.com/JustinAzoff/status/881163562739277824
Because packages tend to be small and many, and each of those has their own dependencies, you can end up with hundreds of packages installed which is simply impractical to manually review.
It is not node, but we do in fact manually review each package we utilize for our given language because it's feasible and worthwhile as the dependency tree is small in this ecosystem. Each and every package is a possible attack vector whether that be intentionally or just because it's poorly written and we can't simply ignore that because it's the done thing and "the community reviews them".
How about package manager managers accept their enourmous responsabilty? urllib vs urllib2, one is a virus? Sorry but that is not "negligence of the developer"
Another easy thing to implement would be a popularity check: "This package was only installed nnn times. Did you mean xxx, or do you want to proceed with the installation of yyy by author dev@g00gle.com?"
Email verification is a must.
PyPI is not a curated distribution.
Also http://evilpackage.fatezero.org/ / https://github.com/fate0/cookiecutter-evilpy-package
That one has neutered the call-home code by now, though.
Package managers need to do more. If they had an enterprise version that you could subscribe to monthly/annually invoice that you would get enterprises onboard, they are concerned about security and will pay. Developers like us will help encourage it. I'd rather not see some third-party "secure" package managers but make them part of PyPi and send funding to the Python foundation. They are seeking donations but that doesn't work well with businesses. Make it a monthly/yearly service.
They solved the issue 2 decades ago by heavily vetting packages before accepting them into repositories. Users are allowed to add and use packages from 3rd party repositories.
Maybe solution to this is creating curated repositories based on publicly open ones and using them by default (and requiring opt-in for using other repositories). Conda for Python and Stackage for Haskell seems like relevant solutions.
Each commit that went into trusty required a team to submit and a team to approve. That costs money. ;-)
Stable/lts/enterprise distributions have other concerns like preventing regressions and configuration or behavior changes during lifetime of release.
Rolling distributions like Arch and OpenSuse Tumbleweed on the other hand can move a lot faster but still provide basic vetting wrt security and sanity of new/updated packages.
https://www.schneier.com/blog/archives/2008/05/random_number...
A.K.A., the Debian openssl Fiasco.
Just one example of distros fucking up the packages from upstream and causing major havoc.
Also the project pioneered reproducible builds and implemented build hardening for most packages.
Arguably. The issue with typosquatting on PyPi has been known and demonstrated for a long time, but nothing has been done about it. Considering there are ways of closing this attack vector, even though it would require some serious work, I'd consider this a bug. It's just a bug that's being exploited now.
For popular packages perhaps, but for many more obscure and niche python packages Ubuntu is often a several releases behind, and that is if there even exists a package to begin with
On the contrary, on distributions that perform security updates the level of security of a package can only increase over time.
It might sound obvious but vulnerabilities are created in new releases, while vulns in existing packages can be only be found and fixed, not created.
(Of course I'm talking only about vulnerabilities here and excluding removal of obsoleted crypto or addition of new security features)
Language package managers solve the problem that we don't have neither an universal package format that works across all programming language requirements and all sort of OSes, nor the time to create an OS specific package for all sort of OSes.
There is nothing wrong with inventing their own solution if they're solving other problems, what is wrong is not learning from previous examples and fucking up creating problems that have been already solved in the process.
Distro repositories are a great example of 'secure for ideal users'. They give you security if you can put up with a small selection of software and older versions. In practice, we end up working around distro repositories by installing stuff with pip, or PPAs, or downloaded from websites.
Yet Amazon and other big tech companies have a very similar process of packaging open source software for internal use and relying on "outdated" libraries.
> I sincerely doubt that Debian can provide adequate security to its almost 50,000 packages.
There's a security tracker where you can see how quickly packages are assigned CVEs and patched - sometimes even before the upstream patch is ready.
Not really, PPAs (and equivalents like copr or obs or aur etc etc) are mostly vulnerable to similar problems. People do want to install upstream software for various reasons, blaming language package managers for the reduced security of that is imho disingenuous.
Is Node is used in government and military solutions? If so then the NPM ecosystem is likely targeted by state actors, and it is a sitting duck.
Wonder what is an effective form of protection against such attack vectors?
Do digitally signed certificates fit into this usage scenario??
No, because either the package author would have to sign them, in which case you have to choose to trust each package author, or the repository would sign them, in which case there would be no improvement for this current issue, since the repo would sign the fake packages as well.
The best way to handle this is whitelists of trusted package maintainers and/or code authors.
I felt a bit curmudgeonly but we have a responsibility at https://qbix.com/platform for all our apps being secure. I wanted to use repos for each package and manually git pull or hg pull them when they changed.
I was finally convinced by our developers to just use package managers with version pinning. Honestly it's really hard to avoid package managers, especially for all the newer functionality such as Payment Requests or Web Push. Luckily there is version pinning.
We want our clients to feel secure that we vetted ALL the code that went into the platform. So our package json (and composer.json) uses version pinning. We'd rather take a bug report and manually fix it than NO bug report and have a SHTF moment.
In your POM file do you have a checksum?
Deployed Maven artifiacts from Central are to required to be signed with a PGP key and are only supposed to come from approved hosts. I don't know how strictly that is enforced and how hard it is to become a host, but at least there is some kind of process.
Maven Central also doesn’t allow the removal of artifacts after they've been published, and every artifact requires a unique version and name. And the names are namespaced. So you don't have the issues that you see with npm, where someone can pull a package and break everything people are using, and then some third party can come in and publish anything under the exact same name.
Is this model perfectly secure? No, you still have to trust that the artifact was signed by a non-malicious person from a host that was not compromised.
When a dependency changes, all the projects that directly depend on it should get notified immediately and their maintainers should rush to test the new changes, to see if they break anything.
There is no shortcut around this, because if B1, B2, ... Bn depend on A1, the consequences may be different for each Bk.
The only real secure optimization that can be done is realizing that some of the Bk use A1 the exact same limited way and thus make an intermediate A1b that depends on A1 which those Bk's depend on. These "projection" builds may be automated by eg the set of methods called by the B's.
Anyway, this is the way that iOS does it before iOS 11 comes out to users. They release a beta to all developers. And they even fix bugs in the beta before releasing to the public.
Without beta testing periods, you can get laziness and just auto-accepting of whatever cane out.
There is be an "alpha release" feature in git where maintainers might put out the next version to be tested by all who depend on it. THIS FEATURE SHOULD NOTIFY THE MAINTAINERS SUBSCRIBED TO THE REPO. THE BUILD ITSELF SHOULD GET ISSUES AND RATINGS FROM MAINTAINERS AS THEY TEST THE NEW BUILD. And releases should not be too frequent.
This is the way to prevent bad things from happening. But that also means that the deeper the dependency is, the more levels this process could take to propagate to end-users.
https://pypkg.com/pypi/xml/f/setup.py
Dork: site:https://pypkg.com intext:"just toy, no harm"