75 comments

[ 2.8 ms ] story [ 142 ms ] thread
(comment deleted)
Reproducible builds are one of three components necessary to have a secure code delivery system: https://defuse.ca/triangle-of-secure-code-delivery.htm

That Red Hat and Ubuntu aren't interested in delivering trustworthy software means you shouldn't trust them.

This isnt accurate. High-assurance established the requirements by mid-1980's in TCSEC. Wheeler, inventor of reproducible builds, also has the best page for SCM security which describes such methods. Here's what it takes:

1. Build system that only developers can access, review of their submissions, a trusted compiler that turns them into binary, and then signs both binary and source.

2. That source or binary is distributed with its signature over any medium. Add TLS for defense in depth if you want.

3. User checks their against signature to determine if trustworthy. Source they compile themselves if worried about substitution.

Now, maybe that repo is compromised. Decentralized checking approach says let them pull source signatures from many other repos that are all building from same source on diverse platforms and such. Go with what (strongest) majority says while reporting inconsistencies. Compile source yourself for sure.

Now, the user might not trust the compiler. If repo is trustworthy, then compiler source should be too with bootstrapping done by experts in diverse ways. However, if one really wants compiler verification, they have to both understand the source and turn it to machine code themselved for at least first run. Ive shown here before how to do that increnentally.

Looking at above, it boils down to a secure, signed distribution of source code for compiler and app. That's all you need. Reproducible builds can have benefits but arent good enough to stop subversion that will mostly be 0-days in source or attacks on endpoint. They're not necessary to stop MITM attacks given signed distribution of source handles that. They're not necessary to bootstrap major compilers as distros come with scripting or executable tools by default that can be trusted to do initial source. If you cant trust them, your distro should already be considered subverted.

There you go. That's about all of it if we're not talking about eliminating code injection from or formal verification of these components. So, strongest repo security they could build should be FOSS's plan. Repo appliances on microkernels or stripped OpenBSD with append-only storage. Signatures (incl key storage) in HSM's or carefully-done embedded boards. Transport node for Internet connection on its own machine with mediation against device-level attacks on host.

Old shit. Would work if they just put work into it. Oh yeah: paper backups hidden in various places. As fail-safe. :)

OpenEmbedded BitBake supports file integrity verification for remote code sources, http://www.yoctoproject.org/docs/2.3/bitbake-user-manual/bit...

File integrity is of key importance for reproducing builds. For non-local archive downloads, the fetcher code can verify SHA-256 and MD5 checksums to ensure the archives have been downloaded correctly. You can specify these checksums by using the SRC_URI variable with the appropriate varflags as follows:

     SRC_URI[md5sum] = "value"
     SRC_URI[sha256sum] = "value"
I love that tools like BitBake support integrity, but I also feel it's not enough to merely support it. We must incentivize it, if not outright require it.

Integrity is a property of the whole ecosystem. Our toolchains are complex, and call other tools, and those call yet more tools. Optional "support" at any layer means even though the user might ask for integrity at one level, their security and integrity may be unverified by another layer of tool. If we want integrity from the whole ecosystem, we must demand it from every part.

Integrity should be a mandate, not an option.

Most (all?) OpenEmbedded production recipes already use this mechanism to specify hashes during builds.
Most isn't enough. "All?" with a question mark isn't enough. Just like one zero-day is enough to totally compromise a system, one unverified source anywhere in a build dependency tree is a great way to land just one RCE, leaving your system precisely as hosed as if you just got hit with a zero-day. :)

I saw another blog about the summit the top link reports on discuss this as well -- from https://garbas.si/2016/reproducible-builds-summit-in-berlin.... :

    > As with zero vulnerability days, there should probably be zero reproducibility days.
Good summary, but disappointing reactions from other distros.

Just to add some info: In the past few days a couple of people gathered in Berlin to discuss reproducible builds. There were some people from the distributions quoted that they don't show much interest, so maybe there will be more movement than this article suggests.

My personal take is that I think there are a few more pieces that are needed for really trustworthy software distribution.

I tried to get a grip on that with the idea that we have a chain "upstream repo" - "usptream tarball" - "potentially insecure transport" - "distribution compile" - "package" - "user download".

Reproducible builds basically fix the tarball to package way, but there's a lot more. E.g. how does a repo become a tarball? Who's checking that? And how does user a know he has the same software as user b? (This is mentioned at the end of the article with the comments of Joanna Rutkowska. Others have discussed basically the same ideas under the term "binary transparency".)

Have you taken a look at OBS (openSUSE)? Every package in openSUSE:Factory should have a link or _source script to fetch the archive which is then used for the build, so the source of the source should be transparent.
Ideally, you shouldn't even upload the sources to the OBS manually at all! Just provide a so called "service" file with an instruction hour to fetch the sources (tarball or git), and everything after that will be handled by BuildService.
I use disabled services. The _service auto-run magic always makes me feel iffy (it won't work on my local machine for some reason, for example).
Sadly once one start down that road on find oneself deep down a endless rabbit hole before one know it.
Tarball downloads by Debian maintainers are secured using OpenPGP and the uscan tool, there are lots of packages where the upstream doesn't do signatures though.

https://wiki.debian.org/debian/watch#Cryptographic_signature...

The Debian maintainers then sign those tarballs before uploading them to Debian.

Verification of the repo -> tarball process is manually done by some Debian maintainers, not everyone does that though.

I wonder if Canonical could use this to their advantage, now that they are 'taking a stand against unofficial Ubuntu images'.[0] (looking at you, OVH...)

It could also go a long way towards addressing concerns that they, as a UK corporation, may now be subject to the Investigatory Powers Act requiring encryption 'back doors' be inserted in communications software.[1] Even if you argue that Ubuntu isn't communication software, they do ship plenty of packages that are.

[0]http://insights.ubuntu.com/2016/12/01/taking-a-stand-against...

[1]http://www.theregister.co.uk/2016/11/30/investigatory_powers...

Is Debian really built on private computers?

How is this any more secure than a random PPA/EXE download from the internet?

What is the alternative? Public computers?
Centralized computers of reputable projects are much more secure than private computers of individual developers
That's what the Debian buildbots do, plus companies doing their own rebuild.

Reproducible builds allow comparing the outputs: they'll have to be identical.

The source ultimately comes from those developers' computers, though. Do you trust those computers not to tamper with the source?

If you do, but you don't trust them not to tamper with the resulting binary - well, that's what reproducible builds are for. If you don't... you've got bigger problems than your build system.

Who reads ALL the source code? If no one, then it doesn't matter where it's built.
> Is Debian really built on private computers?

"Depends". Package maintainers usually build the binary package for their arch themselves (typically x86_64 nowadays). The ~dozen other archs are built by buildd, which also largely runs on hardware* controlled by private people / entities as far as I'm aware.

See eg. https://db.debian.org/machines.cgi

* you can really forget building software on emulators. Way too slow.

What about FPGA-based emulators for building older architectures? Anyone trying that?
While there are a bunch of older archs most are quite well and alive (hence supported by Debian). For example, there are a bunch of POWER8 machines in buildd, or some fairly recent SPARC machines.

Some of these machines are donated. FPGA emulation would be a huge project, both time/involvement wise and also cost-wise (FPGAs are freakin' expensive).

Software emulation gets nowhere near usable performance, even with older archs (say PowerPC970) the real, >10 year old hardware is much much faster than any emulation, which also tend to be rather limited. For example, PPC in QEMU is actually not the 970 (G5), but an G3ish thing. Conversely, the virtual machine is limited to 1.5 GB of RAM (iirc). Technically you can build stuff on that. It just takes ages. 2-10 minutes for compiling a C module. The real deal (a G5 in the cellar) on the other hand can be stuffed with RAM to 8-16 GB and will be about as fast as a slow laptop nowadays is. (Except it will make much more heat and will be louder. Well, the latter depends on the laptop, hellish noises they make...).

(comment deleted)
The other notable thing about deterministic builds is that they are needed for the nix package manager.
This article focus a lot on security, but deterministic builds are also good property to have for other reasons (e.g. it can serve as strong evidence that a new build environment is sound; gives people more trust in the build process).
> Alarmed at the prospect of being unable to trust all 24,000-plus Debian packages, most of which are built on package maintainers’ laptops, the Debian Project has pushed hard over the last three years to make as many packages reproducibly buildable as possible. As of this writing, 91 percent have achieved that goal, and the project hopes to hit 100 percent in 2017.

91% already? That's pretty impressive! I remember when they first started discussing this but haven't tracked the progress at all. Happy to see how far long it's come.

> most of which are built on package maintainers’ laptops

Gracious, is that really true? My experience of interacting with Debian/Ubuntu is via launchpad.net, where you push the dsc (source tarball + patches + metadata), and the server takes care of producing binary archives for the supported architectures.

For the most popular architecture amongst developers (amd64 right now, i386 earlier), that was true until the last couple of years where Debian supports source-only uploads.
Ubuntu does source-only uploads. Binaries cannot be uploaded by anyone; everything is built on Launchpad's infrastructure (managed by Canonical).

Debian traditionally has done source-and-binary uploads for the maintainer's architecture (usually amd64), and builds for other architectures via the its buildd infrastructure (managed by contributors, AIUI). More recently it is possible to do source-only uploads to Debian now, but I'm told it doesn't work for new packages.

Seems like package hashes could be stored in a block chain to prevent modification by certain three letter agencies and other non-benign actors.
That's basically the idea of binary transparency (which is a bit stalled right now, but people are aware that this would be a good move). BT is a complement to reproducible builds, not a replacement.
Well, yeah, obviously you need a deterministic build before you can have a meaningful hash to store on your blockchain.
Disagree :) We don't need repro to start logging what we build! We should start logging what we build so it's an incentive to get repro results :)

Reproducible builds are a pursuit. You can never "prove" yo have them: only keep checking and be vigilant. So it's best to start logging now, and then keep putting in the work to make sure it's "meaningful".

From [0] > Of course building everything yourself generally isn't practical.

I've been using gentoo for a decade and I'm sad to say that I still can't get to 100% built from source. There is always some useful program out there, or something that I don't want to maintain the build process for, that ends up being a binary that I know nothing about.

Getting to 90% is possible. 100% is extremely difficult requires making some hard choices about what software you are going to use.

0. http://sobersecurity.blogspot.co.uk/2016/05/trusting-trustin...

It's all kind of moot anyway if you are running hardware with blobs (Intel ME, for example). Getting truly to 100% is pretty far away, as frustrating as that is.
With major effort, ME has been disabled on Lenovo X230 (Ivy Bridge).
Where do you get your information from? AFAIK, it's only has been stripped down to a bare minimum, putting it in some sort of recovery mode - but it can't be erased from the BIOS chip yet.
I was thinking of this project - https://github.com/corna/me_cleaner This is the one that strips most of the ME (which seems to be Java applets, if I remember correctly Igor Skochinsky's presentation), but there's still the core "OS" in PCH, which loads and executes the applets - we don't know what nefarious things that part of ME might be up to - and it has DMA, and access to the network controller - that's how they remotely wipe HDD's, even without an OS being installed, even without the computer being turned on - that shit is straight out of Orwell's 1984. Telescreen, that's what it is.
Thanks for the info.

Is there security benefit in running on a desktop with non-Intel (e.g. Broadcom) add-on NICs and nothing connected to the integrated Intel NIC?

Yes. It seems this only works with Intel cards(wifi or ethernet); I don't know the reasons why, though - it might be some firmware support. Also, I find it odd that it can work with the wireless chip - shouldn't the laptop be associated with the AP? If somebody knows, I'd be happy to learn about it.

Source: https://software.intel.com/en-us/blogs/2013/08/07/intel-vpro...

The ME/AMT is the nastiest of them all. And those are not the only ones. FSP, the CPU microcode.

You can get rid of AMT if you're on older hardware, or your willing to buy a laptop from (US) http://www.libiquity.com/, (UK) https://minifree.org/ or (EU) https://tehnoetic.com/;

The future is bleak though, there's not much hope in sight freeing anything from Nehalem onwards. I have a Thinkpad X201, and flashing coreboot with blobs is the best I could for the moment.

I do remember though that I had Gentoo 100% from sources a few years back. But updates were too much of a pain in the ass, and after tweaking the compiler flags, I always ended up breaking my system. So I'd take reproducible builds over that.

"Getting to 90% is possible. 100% is extremely difficult requires making some hard choices about what software you are going to use."

That's what Roger Schell, one of INFOSEC's founders, used to say when pushing his GEMSOS security kernel (now at Aesec). He said you can get quite a bit there with review and hardening of these mainstream codebases. 90% maybe in best case. The last 10% might be impossible if it wasnt designed for high security and verifiability in mind from beginning with it baked into every aspect of it.

Once you go full tin-foil hat, you want silicon level validation. And when you find out that isn't really viable, you realize that an open source driver that uses high-level silicon functions isn't all that different from a binary blob.
There's a really interesting open hardware project that's developing a Raspberry Pi competitor, "LowRISC".

If they end up fully blob-less (no firmware blobs, open chip designs, open board layout) they're going to be the first group to achieve that ever AFAIK, so I'm on the edge of my seat for it.

I'd heard of lowRISC, didn't really realize that was their goal. That would be phenomenal.

Also, I have to ask: is your username a Quantum Thief reference?

Yes, it's a very interesting project. I'm already subscribed to their list, I got on their site accidentally. If they do not succeed, the next best thing is to wait until ARM gets powerful enough, and hope they don't go down the orwellian route, like Intel and AMD.
That's approaching a chicken-and-egg problem. Other parties are working on verifiable open hardware, but their work is only worth so much if any useful software can be tampered with at multiple other points in the chain.

Reproducible builds are an absolutely achievable way of adding trust to that particular step and reducing the overall attack surface.

We don't need to live in a world of two extremes, 'curl http:// | sudo bash' or hand-built silicon.

The idiotic, reductionist viewpoint that "I have intel ME on my machine so anything and everything that you could ever do is worthless and pointless and moot, trust me on that" is so utterly backwards, such an insane state of denial -- I can't even comprehend it. Until I realize it almost exclusively, in my experience, comes from people who eat tinfoil and have never worked in security.

Honestly, I consider it lucky that the people who typically promote this nonsensical bullshit are probably not actually security engineers or researchers, or allowed to do security related work in any way -- because they would be terrible at it with a position like that. Actual security engineers understand that there are things called "threat models" and "tradeoffs" and you can, in fact, measurably improve security in meaningful ways for many systems.

You might as well just starve yourself to death because -- hey -- eating is pointless when you could get hit by a bus at any moment.

Why prevent your application from having SQL injections? Why not just dump all your customer records and private keys on pastebin? After all, the NSA can just steal those secrets from your computer with some Unicorn Magic, so clearly securing anything is actually pointless.

Do you believe reproducible builds cover an important threat to make them worth the effort just for that? As far as I understand - not even a little. But the effort is much more important for the future of package management.
This is a bit unfair. The threat that source auditing is supposed to mitigate is backdooring by well-funded attackers, and it (AFAICT) isn't very effective against those when they can just go down a level. Are there plausible attackers that really stopped by compiling from source, that can ship a malicious blob but can't ship malicious hardware?
Reproducible builds is an excellent QA initiative, but let's not overestimate their security benefits.

They're maybe going to protect us from random ELF viruses on developers' machines, but not from a targeted attack.

Why bother pwning binaries when it's much easier to pwn the source?

I mean, it's so easy it's already been done:

https://lists.debian.org/20030213165225.GB7379@azure.humbug....

Lets also remember that its threat model is FBI, CIA, and do on. They mainly exploit vulnerabilities in software or bad configurations. They're quite successful. Repeoducible builds with normal Linux environments isnt going to stop them. Doing it with "hardened" OS's might but incurs the risk that such niche OS gets less security review.

So, this post and effort is ultimately misguided if it's about securing computers from nation states. Linux cant be trusted for this job. They need strong, endpoint security with verified source and compilation for that. Likely be console apps with a microkernel OS since labor involved will demand simple system.

Also, lets not forget that it's not just Linux (or any one kernel or OS) that's the issue but also every popular hardware architecture can be expected to be subvertable, especially x86 it would appear.

And exploiting these types of bugs is within the capablilities of many non-nation state actors as well at this point.

Oh yeah. It's why you see me pushing things like the GPL'd Leon3 processor (SPARC) that can be implemented on visually-inspectable nodes. Or the 8-16bitters or Forth CPU's if it really needs minimalism. Even open-source EDA tools like Qflow that could use attention from smart OSS contributors.
I can't stand this "throw up our hands" reaction to nation-state adversaries. None of these APTs have unlimited budgets; sufficient hardening and defense-in-depth will encourage the planner to use different targets to satisfy the tasker's intent.

Every security improvement helps. We can't just say "run SELinux in a VM and use Tails with JavaScript disabled or accept the Mossad completely owning your life." There has to be a realistic middle ground.

It's not a throw up your hands reaction. The Linux stacks, from kernel to key apps, have had boatloads of vulnerabilities since they accept insecure code written in languages easy to screw up by people who arent security engineers. Such a stack has no business being used in anything designed to resist well-funded or determined attackers. Federal government's own criteria rates Linux at EAL4: "can protect against causual or inadvertant attempts to breach security." Whereas, even trying to stop nation-states requires very-rigorous developmenf plus review of vastly simpler systems. Systems done like that resisted NSA penetration during evaluation.

So, the trick for FOSS is to do their stacks like that. They refuse. Genode is at least architecturally doing it right. Muen and seL4 are separation kernels that might be turned into the foundation. The pieces are there and could be turned into a usable system with a tiny amount of effort compared to what's going into Linux right now. It can even run Linux apps user-mode. Commercial implementations already exist for these (eg Turaya Desktop, Dell SCC) while FOSS just ignores it like most high-security techniques.

Google Nizza Secure Systems architecture pdf for an example from CompSci. Genode evolved from it but basis could be done in other projects.

I am blown away at the amount of terms you mentioned that I never heard of, despite working in vulnerability research/development for years. I've got a lot of Googling to do. It sounds like you are an expert at secure operating systems.

I want to emphasize that computer network security is more than a secure operating system. It is defense in depth: the combination of well-designed networks that are hard to traverse, isolated endpoints for internet-facing products, management interfaces that are also properly isolated, monitoring that alerts with a low false positive and negative rate, sufficient logging and analysis that surfaces anomalies while keeping noise down, etc. These are hard and should be scaled to the appropriate threat level to prevent impact to usability.

It is possible to keep a computer network secure from APT even with a vulnerable operating system. The APT has to deal with everything mentioned above even if it has spent millions on vulnerability research on a vulnerability-ridden operating system.

"It is possible to keep a computer network secure from APT even with a vulnerable operating system. The APT has to deal with everything mentioned above even if it has spent millions on vulnerability research on a vulnerability-ridden operating system."

All the big companies seem to be failing at that given the TB of data exported. It's pretty hard. Whereas, systems immune to code injection or catching leaks by default might have reduced it considerably. That's why INFOSEC's inventors focused on designing those.

"I am blown away at the amount of terms you mentioned that I never heard of, despite working in vulnerability research/development for years. I've got a lot of Googling to do. It sounds like you are an expert at secure operating systems."

The reason is that INFOSEC the profession acts in almost a pop culture way where what's popular from certain parties goes. Very fad-driven. It was originally created by engineers and computer scientists that did things in a rigorous way with excellent results in pentesting. The strongest form we called high-assurance security or real security if you ask me. They solved tons of problems early on. Later people ignored them entirely starting around the 90's onward. Keep reinventing their lessons but most steadfastly refuse to read & follow their work.

In any case, send me an email if you want a curated list showing where it came from, what was achieved, with what methods, and so on. I'll send as much or at little as you can take. Meanwhile, here's my own framework I posted for free after high-assurance market went nowhere outside defense. It embeds some of what I learned from them.

http://pastebin.com/y3PufJ0V

For interim, I'll give you just a few on the main model that achieved real security: designing system around a limited, strong component where all trust is placed called a TCB. First is paper on first systems certified for security claims based on high-assurance methods invented in 1970's. Both are still available commercially. Second is example of secure VMM (for OpenVMS) done by legendary engineer & INFOSEC co-founder Paul Karger. Pay attention to "Layered Design" & "Assurance" sections then wonder why mainstream INFOSEC is barely securing VMM's today despite techniques (eg leak mitigation) being ancient. Third is architecture that evolved from them in academia that I link since the paper explains the concept really well. Fourth is the only FOSS implementation of that architecture using best-of-breed components like seL4 or Nitpicker. Fifth is Nitpicker just so you see how our sub-field of INFOSEC turns hard problems like trusting X/KDE/Gnome apps into simpler ones. Sixth is a secure, browsing architecture that shows both a way to do it better & how systematically people in high-assurance solve these problems.

http://www.cse.psu.edu/~trj1/cse443-s12/docs/ch6.pdf

http://www.cse.psu.edu/~trj1/cse543-f06/papers/vax_vmm.pdf

https://os.inf.tu-dresden.de/papers_ps/nizza.pdf

https://genode.org/

https://os.inf.tu-dresden.de/papers_ps/feske-nitpicker.pdf

https://www.usenix.org/legacy/event...

> "Red Hat has never intentionally inserted a back door in our products, and we have never received a court order to do so,"

Sure. But that doesn't mean it doesn't contain any backdoors. RedHat (also Fedora, Ubuntu, and many others) kindly accepts non-free binary blobs (in Linux kernel), which may make the system vulnerable with backdoors.

Not much people care about software freedom. Most care about "open source." Though it does have some interest in freedom (ie, opening the source, but not freeing), that's not its vital aim. For most, it is money (though they say, quality), and for a few it is power.

As long as the majority of the world focuses on money or power, we will see some weakness in security, and freedom (That doesn't mean free software are more secure, but the user have the choice).

  Small details like differences in timestamps or build directories result in different binaries.
Why is this even a problem? Did people actually think it was a good idea to have a compiler's binary output depend on incidental crap like this, or did it sneak in somehow?
I am also surprised a compiler includes this information into a binary. My expection is that the compiler creates an optimized machine code functionally equivalent to the source code, and nothing else.
I think a lot of times it just sneaks in. For example, when one of my projects was in alpha state, I added the build timestamp to the `--version` output, so that I could reserve version number changes for actual releases. At that point in time I never thought it would become a debian package, much less that reproducible builds would be a thing someday.
There are many ways to intentionally compile this kind of information into your binary. It's frequently done to include things like "this binary was compiled on 'x date'", usually for debugging use.
It's also worth highlighting that diffoscope.org is an extremely cool tool. This project is worthwhile just for having encouraged its creation.