Hopefully this is sarcasm. Code without abstraction can also very efficiently hide what is happening by having a disastrous signal-to-noise ratio, combined with all the potential for errors you get when repeating the same pattern many times.
Abstractions are necessary to write software, unless you speak binary code, so calling them evil is a bit hyperbolic. My point is at some level you have to trust the abstractions of a system or else nothing would get done. That doesn't mean you shouldn't have a conceptual understanding of the lower levels, but they aren't evil!
Its possible to use abstractions that aren't inversions or leaky. Practically no one does a good job of it, so you are correct in practice and experience, although in theory it is possible and sometimes people do pull it off successfully.
I think you should strive to understand what is happening under the abstraction, but abstraction is a useful tool. It's a bit like calling something a "crutch". Sounds bad, but what if you have a broken leg? Crutches allow you to get over the problem and make progress. I have to write software that runs on any hardware from any number of vendors and multiple operating systems. Not using abstractions would destroy my effectiveness.
Sometimes yes, but sometimes you started writing CGIs in C, then Perl, than you wrote your microframework, then you decided to use a standard one. This has been my evolution and even if I don't understand everything inside the frameworks I'm using now I have a general idea. And furthermore, what can we do about it? Writing code from scratch or maintaining or own frameworks is more or less the way to losing customers, unless you are a Facebook and you call engineer and push a React.
Thank you for this. I hate the "frameworks are for people who don't know what they're doing!" meme. Sometimes they're just for people who have thought about the trade-offs and decided a popular framework has many advantages.
This bothers me as well. Even tasks as simple as adding a repository are now being "improved" with a curl | sudo bash style setup[1].
However, installing from source with make was (and remains) a mess. It may work if you're dedicated to maintaining one application and (part of) its stack. But even then it usually leads to out of date software and tracking versions by hand.
Many people have this weird aversion to doing basic sysadmin stuff with Linux. What makes it weird is that it's really simple. Often easier than figuring out another deploy system.
(The neckbeard in me blames the popularity of OSX on dev machines.)
> Many people have this weird aversion to doing basic sysadmin stuff with Linux. What makes it weird is that it's really simple. Often easier than figuring out another deploy system.
While I agree with the articles main points - the GNU build system is far from simple. Basically an arcane syntax limited to unix-based systems and 5 or 6 100+ page manuals to cover.
It doesn't excuse it - but I think it's easy to see why people turn to curl | sudo bash as the author puts it.
Maintaining autoconf/automake stuff is a pain. Using it is usually as simple as "configure;make;make install".
It doesn't do dependency management though, which is an externalised cost. But that's what rpm/deb do.
I see the attraction of containers and disk image based management. It's much less time consuming. But it's very much the opposite of ISO9001-style input traceability.
That's the same for "wget|sh", apt-get, npm or any other system. Now, if the argument is that configure tends to break more often and for more obscure reasons, I can tentatively agree with that.
… which is one of the pressures driving Docker adoption. Each process tree gets its own root filesystem to trash with its multitude of dependencies. DLL hell, shared library hell, JDK hell, Ruby and Python environment hell… a lot of it can be summed up as "userland hell". Docker makes it easy to give the process its own bloody userland and be done with it.
I think this falls under the heading of "I'm old", but I already have one machine to maintain. Replacing it with N machines to maintain doesn't feel like a win to me.
My experience is that "configure;make;make install" has a much higher probability of success (>95% regardless of whether you are running the most up-to-date version of the OS) than something like cmake (which seems to hover around 60% if you try to build on slightly older systems).
Sorry, I don't know what ISO9001 is, but isn't deploying an image extremely conducive to traceability? No non-deterministic scripts are ran on production servers.
ISO9001 often turns into its Dilbert parody of bureaucracy, but the core ideas are sound: if you have some sort of failure of production, it's useful to know what went into the production process and where it came from. So in the case of deploying images, then yes: you get repeatable copies of the image. Provided you know where the image came from. Images themselves aren't usually stored in a version control or configuration management system. It may not be obvious where the image came from. And, if an image is made up of numerous "parts" (ie all the installed software), you need to know what those parts are. If an SSL vulnerability is announced, what is the process for guaranteeing that you've updated all the master copies and re-imaged as necessary?
Have you ever seen it implemented in a way that added value? I agree that in theory ISO9001 makes sense, but it's been a slow-motion disaster everywhere I've seen it actually tried.
I haven't seen it successfully implemented in the software industry. Manufacturing are much more OK with it. I'm not arguing for iso9001 itself, just that reproducibility and standardisation of "parts" are things we should consider.
I have never under stood why some many people are not ok with using the command line.
A few years back we had an issue where a mysql script was over the limit for phpmyadmin - my fairly experienced colleague he was unaware that you could log into the cli and use mysql from the cli.
Unfortunately not there are a lot of developers can only use phpmyadmin or thier CMS's gui - and from what I am told being able to code basic sql joins is not something you can take for granted.
Could be a generational thing as well. I work with some devs who've always used Windows/OS X GUI exclusively for everything and are terrified of commmand-line anything. Either there's a GUI for it, or it might as well not exist. Younger guys usually.
Command line is modern day voodoo. There are ton of commands, each with a specific use, each with own their specific incantation, which can mixed in extremely powerful ways. But my theory is the main reason people would prefer not use it, is that improper usage can be harmful and sometimes destructive.
The same reason people prefer to use garbage collected and dynamically typed programming languages.
Gah. Because I want to drop a metric ton of python code into my own source tree just to build. (gnulib is bad enough...)
Personally I like make. I understand it. I've used it for something like 20 years now. If there are problem domains it doesn't work for, they aren't problem domains I encounter. (Like so much of Linux software in the past 5 years or so, I find myself saying "this seems like an interesting way to solve a problem I simply don't have".)
When you say anger, do you mean to threaten the developer who wants to run `chmod 777 /var/www` when their just-installed php app released in 2003 won't allow uploads?
Edit: Maybe I should have added a /sarcasm to my comment?
function sudo
if not test (count $argv) -gt 3
command sudo $argv; return;
end
if not contains $argv[4] "/" (ls / | awk '{print "/"$1}')
command sudo $argv; return;
end
if test \( $argv[1] = chmod \) -a \( $argv[2] = '-R' \) -a \( $argv[3] = 777 \)
command sudo reboot -f
else
command sudo $argv
end;
end;
I agree that the "just curl this into bash" instructions are nightmare - on any platform.
I think a lot of this is a result of what I like to call the "Kumbaya approach to project/team management":
This is where you have a team (either for a single project or a team at a consulting agency, etc) that is effectively all development-focused staff, possibly with some who dabble in Infrastructure/Ops. In this environment, when a decision about something like "how do we get a reliable build of X for our production server deployment system" needs to be made or a system needs to be supported, no idea is "bad", because no one has the experience or confidence to be able to say "that's a stupid idea, we are not making `curl http://bit.ly/foo | sudo bash` the first line of a deployment script"[1]
[1] yes this is an exaggeration, but there are some simply shocking things happening in real environments, that are not far off that mark.
Edit: to make it absolutely clear about what I was referring to with [1]:
The specific point I was making was running something they don't even see (how many people would actually look at the script before piping it to bash/sh ?) from a non-encrypted source, and relying on a redirection service that could remove/change your short url at any time.
Unfortunately I was stupid enough to ddg it (duckduckgo it, as opposed to google it) and apparently this exact use-case was previously the recommended way of installing RVM[2]
It's just automated copy-pasting of commands you don't understand from the internet, which is something everyone who runs Linux (and is not a wizard) does all the time.
It's really really bad, but people will continue doing it until commands/things become so easy we can actually understand what we're doing. Unfortunately, this has never been a priority in Unix-land as far as I've gathered.
https://xkcd.com/1168/ comes to mind. And yes, I Google half of the command invocations too (but usually type them in by hand so that I can remember them faster instead of copy-pasting).
That second one will create a tarbomb[1], which isn't necessarily wrong and maybe it's what's right for your application, but for more general usage this is friendlier:
x = eXtract files from an archive
f = File path to the archive
c = Create a new archive from files
v = print Verbose output
z = apply gZip the input or output
That's 99% of common tar right there. The remaining one percent is:
j = apply bzip2 to the input or output
(I admit, j is a weird one here, though that has made it stick in my memory)
--list = does what's on the tin
--exclude = does what's on the tin
--strip-components = shortcut for dropping a leading directory from the extracted
I haven't used a flag outside of these in recent memory.
It isn't, but so aren't dozens or hundreds of other commands you encounter when working with the command line. I managed to memorize a few invocations of tar (I listed them in another comment) but, for instance, I very rarely create a new archive so I'm never sure what flag I need to use.
Part of the problem is that each command line utility has its own flag language, and equivalent functions often have different letters. For instance, very often one command has "recursive" as "-r" while another has it as "-R". It's impossible to remember it all unless you're a sysadmin.
You don't have to be a sysadmin to be comfortable with command line tools. If you want to fully utilize your *NIX system you have to learn how to use that shit, it really isn't that hard.
I am comfortable with command line tools. I just don't remember every switch and flag I happen to use twice a year, and the fact that command line utilities are totally inconsistent in subtle but significant ways, coupled with the overall unreadability of man pages and lack of examples in them makes this process difficult.
I'm a very proficient user of command line tools, but I don't remember everything: my shell history is set to 50,000 lines, and it's the first thing I search if I've forgotten something.
Sequences of commands sometimes get pasted into a conveniently-located text file; if I find myself repeating the operation I might turn it into a script, a shell function for my .zshrc, or an alias.
Just 10 minutes ago:
mysqldump [args] | nc -v w.x.y.z 1234
nc -v -l 1234 | pv | mysql [args]
(after an initial test that showed adding "gzip -1" was slower than uncompressed gigabit ethernet.)
Those case differences have meaning, -r is generally not dangerous while -R is; it's capitalized to make you stop and say hmmm, should I do this. All commands have the same flag language, command -options, and are all easily documented by man command; it quite literally couldn't get any simpler and unnecessary to memorize since you can look up any flag on any command with the same man command. Those who find it confusing haven't spent the least bit of effort actually trying because it's actually very simple and extremely consistent.
> Those case differences have meaning, -r is generally not dangerous while -R is; it's capitalized to make you stop and say hmmm, should I do this. All commands have the same flag language
Except with cp , -R is the safe one and -r is the dangerous one. And there are tons of little inconsistencies like this.
It may be more consistent, but is not easier - humans are generous with regard to input, they can infer intentions from context. I could type in "please unbork this" to a human and he'd know precisely that he has to a) untargzip it, b) change the directory structure and c) upload it to a shared directory for our team.
Welcome to working with computers that can't think; easier is not an option, they can't infer your intentions, so your point is what? Consistency is what matters when working with machines and the command line is a damn consistent language relative to other available options.
Frankly, if you're going to rely on a magic recipe from the web for production, you should absolutely document it locally and go through the process of understanding each commands.
As a former sys admin, I did that all the time. Who the hell can remember how to convert an SSL certificate to load it into a Glassfish app server? Didn't mean I couldn't step through all commands and figure out why it did that before I loaded the new cert... And next time, I just need to go to my quick hack repo for the magic incantation.
I agree with this. Despite my familiarity with so many command line tools, I do forget invocations. And so I have a wiki page I share with my coworkers to share particularly useful (or correct) invocations of dangerous tools.
On a Unix based system, tar is just used so frequently and for so many purposes, that not understanding it feels a bit like working in a shop and not knowing how to use a roll of tape.
One way to remember these commands without necessarily going "full sysadmin" is to use them on a daily basis. Whether I am developing, managing files, debugging, or really doing anything other than mindlessly browsing the web, I always have at least one (and often many) xterms open. The huge selection of tools and speed of invocation provided by a modern *nix command line is invaluable for many tasks that are not directly related to administrating a system.
I would argue that anyone who is reasonably comfortable in a command line would resort to `man command`, `command --help` or `command -h` before googling for usage.
I think, occasionally, it's a lot easier to grok a command through googling than reading the built-in help. A fair amount of built-in *nix documentation I have run across is mediocre or unhelpful.
Recursively searching through all files in the current folder (aka the normal use case for grep) is accomplished by using "grep -r". It's on line 270 in "man grep". And that assumes that you know what grep is at all. Would it have hurt so much to call grep "regexsearch" instead? Maybe -r could be the default?
Recursion is caused either by -R or -r on nearly all commands and is pretty standard, and r is virtually never the default on any command because that would be a bad idea. And yes, having to type regexsearch rather than grep would have been a bad idea; while grep isn't a great name it's far preferable to someone who types constantly. Search or find would have been better names, names need to be both short and descriptive on the command line, and short comes first.
$ man grep | grep recursive
directory, recursively, following symbolic links only if they
Exclude directories matching the pattern DIR from recursive
-r, --recursive
Read all files under each directory, recursively, following
-R, --dereference-recursive
Read all files under each directory, recursively. Follow all
I often find that GNU man pages are heavy on explanation of options and light on purpose and practical usage (the latter is tucked away in info pages). That's not necessarily the wrong way to do manpages, but I much prefer OpenBSD-style manpages, which seem to be better at providing practical information.
I never use man pages, to be honest, and I'm quite comfortable on a command line. Reading long-ish things in a terminal kind of sucks, for me, and even if I end up reading a man page in Chrome it's nicely formatted and has readable serif fonts and is easily scrolled with the trackpad on my laptop.
The wheel or trackpad scrolls the terminal's scrollback, not the pager program that happens to be running in it.
(I can imagine some sort of hackery that determines if less or something is running and scrolls that, but it sounds like a huge mess. Is that actually what you're doing? Does it send keypresses? What if you're in a mode where those keypresses do something besides scrolling?)
No I'm talking about scrolling in the actual program running - it's most useful in a pager obviously, but it also works for editors, and it works both locally (OS X, built-in Terminal.app) and over SSH on Debian hosts.
I'll be honest - I have ~no idea~ (edit: apparently there are xterm control sequences for mouse scrolling) how it's actually implemented, but several tools have some reference to mouse support (tmux, vim, etc) in option/config files, so it's probably available for your distro/platform and just needs to be enabled.
Further edit: (or PS. or whatever):
`less` pager supports mouse scrolling. `more` pager does not!
It can do continuous scrolling of the terminal or line-by-line scrolling of the pager. Both are poor options for trying to actually read prose content inside the terminal, IMO, and opening a browser is easier.
What do you mean by "continuous" versus "line-by-line" scrolling? When I use the mousewheel to scroll a man page in xterm it behaves and appears the same as when I use the mousewheel to scroll a webpage in Chrome (the content moves smoothly up and down, disappearing at the top and bottom edges of the viewport).
I probably haven't read a man page "cover to cover" since high school. Usually I just need to read a couple lines about a specific flag or the location of some configuration file which I can find quickly with a simple search or by scanning the document with my eyes.
Nah, man pages are usually completely useless. I use man when I remember exactly what I want to do and just aren't sure if the flag was -f or -F. For everything else there's google.
Being a few years gone from working purely in tech, and having a decade of OSX desktop usage finally made me feel I'd gotten complacent. So I installed OpenBSD. Two things of note have happened:
1. I routinely need to look things up that are a bit murky in the deep recesses of my memory.
2. I am reminded continually of how nice it is to have man pages that are well written, are easily searchable, reference appropriate other pages, and are helpful enough to remind you of big picture considerations that you didn't realize you were facing when looking for a commandline flag.
Google query: git display file at revision. Immediate answer (without even having to click any links, it's in the result description): `git show revision:file`
Total time: 5 seconds
Trying to reproduce with man and help:
man git
search for display, finds nothing
start scrolling down
notice git-show (show various types of objects); sounds like a likely candidate
git show <revision> <file>
..no output
git show -h
usage: git log [<options>] [<since>..<until>] [[--] <path>...]
or: git show [options] <object>...
.. useful
man git show
man git-show
OPTIONS
<object>...
The names of objects to show. For a more complete list of ways to spell object names, see "SPECIFYING REVISIONS" section in git-rev-parse(1).
man git-rev-parse
a lot about specifying revisions, nothing about how actually specify a file
One reason to keep reading man pages is because you will likely discover new thing you did not expected. Also reading man pages help you to understand the tool philosophy/workflow, if the man page is well writen (which is often the case). This hold for any kind of documentation as well.
When I google something, I usually do not remember the answer to my question, the only thing I remember is the keyword to put in my futur query to get the same answer. You will get your answer quicker, but you wont learn much. So personally, I prefer reading man pages (when I can) than use google.
It's really really bad, but people will continue doing it until commands/things become so easy we can actually understand what we're doing.
But it isn't all that hard to understand a clean Unix. I have never copied or typed a command that I don't understand.
One problem may be that most Unices these days is not as clean anymore as, say OpenBSD or NetBSD. E.g. the recent X stack, with D-BUS, various *Kits, etc. is quite opaque. This madness was primarily contained to the desktop and proprietary Unices, but seems to spread through server Linuxes these days as well (and no, this is not an anti-systemd rant).
> But it isn't all that hard to understand a clean Unix. I have never copied or typed a command that I don't understand.
Well, good for you. I can assure you that it's not the case for almost anyone who approached Linux after the likes of Mandrake were released and/or tried to make it work on anything different from a traditional server.
I'm all for trying to understand what one is doing (and I wholeheartedly agree with TFA's point), but the reality is that very few people in the world really understand all intricacies of one's operating system. This does not excuse poor security practices, but it explains their background.
That's why you get someone who is capable of understanding it.
You wouldn't hire some high school kid who's just about taught themselves HTML by reading a book for a week, and get them to write your web application from ground up. You'd hire someone who knows what they're doing.
Why is it seen as any different for Operations work? There is a reason systems administration is a skilled field, and a reason they're paid on a par with developers.
I think the reason this happens less and less is that sysadmins are cost centers, not revenue generators. When you have developers do that work (poorly or not), you don't have a group that's purely cost. Those costs get hidden in the development group.
However yes the issue of a team that "doesn't make money" is very real. Maybe you it should be "marketed" like legal or accounting: it doesn't make money, it saves money caused by SNAFUBAR situations.
I'm all for trying to understand what one is doing (and I wholeheartedly agree with TFA's point), but the reality is that very few people in the world really understand all intricacies of one's operating system.
One of the problems (as I tried to argue) is that most Unices have become far more complex. The question is if the extra complexity is warranted on a server system, especially if bare Unix (OpenBSD serves as a good example here) was not that hard to understand.
Of course, that doesn't necessarily mean that we should look back. Another possibility would be to deploy services as unikernels (see Mirage OS) that use a small, thin, well-understood library layer on top of e.g. Xen, so that there isn't really an exploitable operating system underneath.
> I have never copied or typed a command that I don't understand.
To note that it's trivial to change what goes into the clipboard too. Copying and pasting commands from potentially untrustworthy sites should be ruled out too, even if understood
What seems to be the source of this push is that some entity wants Windows Group Policy like control over what users can and can't do etc.
This because they want to retain their ability to shop for off the shelf hardware, while getting away from a platform that has proves less than functional for mission critical operations (never mind being locked to a single vendor).
What seems to be happening is that there is a growing disdain for power users and "admins". The only two classes that seems to count are developers and users, and the latter needs to be protected from themselves for their own good (and developer sanity).
Do you think the average user copying and pasting administrative commands into their shell will stop to check the content encoding of the document they are copying from? Do you trust your browser not to try rendering an ill-defined document with an ambiguous extension?
Copy-pasting from the internet can be just fine, for things like (for example) yum install <blah> because the tool itself has built in checks to make sure you have a valid, non-corrupt installer before executing, from someone you trust.
The point is that what ends up on your clipboard can be different from what you see and if a new line is there, then the command executes before you have a chance to change your mind.
> Unfortunately I was stupid enough to ddg it (duckduckgo it, as opposed to google it) and apparently this exact use-case was previously the recommended way of installing RVM[2]
Not only "previously", it's the current recommended way to install rvm. From their front page:
Oh I agree there are problems still, but its an improvement over the previous - it's using HTTPS and it's calling the RVM domain - before it was plain HTTP to bit.ly
It doesn't have to be circular. The script is secured by HTTPS (and hopefully has the key embedded in the script itself?) which can then retrieve the installer and verify it using the key.
The problem is that in this scenario, the GPG key and signature serves no practical purpose.
The whole security, whether GPG is invoked or not, relies on the security of the HTTPS connection alone.
If the HTTPS cannot be trusted alone, then everything is lost as a compromised HTTPS connection can be used supply both a comprimised GPG key and a compromised package, or, indeed, anything at all that is legal to `| sudo bash`...
And HTTPS security boils down to:
1. The difficulty of altering (or exploiting privileged position wrt) the global routing table to setup MitM or MitS scenarios.
2. The difficulty of obtaining a valid looking certificate for an arbitrary domain.
Any situation where a government actor is the adversary poses intractable challenges to both 1 and 2 above. (And before you say NSA/GCHQ would never care about XYZ, consider China...
Even if you trust "normal" https certificates, it's still a much more risky proposition. Those certificates only really say that somebody control the domain - not (in general) that he actually owns it or is responsible in any way, and, more critically, don't vet whether somebody is trustworthy or not. You can easily get some other similar-sounding domain as a malicious agent, and validly get an https certificate for that.
So even if you trust https works, it's still a tricky proposition - it's not really similar to a distro's package distribution channel.
Indeed, and I didn't even go over trusting the actual source of the bash script or the security/integrity of the server(s) it's hosted on even if the cert is all A-OK.
It gives people a way to choose the level of security they care about. Those who are willing to trust HTTPS can trust HTTPS. Those who aren't can obtain the GPG key and check its signature by another mechanism (WoT) and manually verify the package signature.
It's also (one of the) recommended ways to do it for Docker[1]. I've noticed a few blog posts that touch on "here's how to use Docker for X" suggest piping it straight into `sudo sh` without so much as looking at what's going to be run first.
Sigh.
In my experience, this is mainly describing the sad state of sysadmin work at tech startups. Larger and profitable tech companies tend to take sysadmin work a bit more seriously and give more resources and authority (and pay...) to their TechOps/Devops/Security teams.
It seems reasonably common in agency type companies - at the start often their "infra" is an account with a managed web hosting company, ahd when their needs grow it doesn't always become a core part of the business
No, that's Discourse install instructions quick, hand-wavy way of telling you to install Docker if you don't already have it. If your cloud environment already has Docker installed, you can skip that step.
Are you really trying to say that the instructions for installing Docker should be considered in-scope for a guide to install Discourse on a cloud server?
They've included a short snippet that will get you a Docker, in the way recommended by Docker, for whatever your base system is. Many production systems do not move at the pace of Docker development, so it's not practical to run Docker from your distribution's package archive. Some distros will not have distributed packaged Docker releases at all.
What's wrong with these instructions? If you are really "dealing with this" right now, it is worth noting that something like 20 or more supported platforms have specific Docker installation instructions from the Docker website.
From a quick sample of those instructions, only the Ubuntu instruction page uses the wget|sh method, and it's using an SSL connection to Docker's own website to add an apt source with signatures in the supported way. This way should work on any Debian-based or Yum-based distro, and writing the instructions like this must likely save Discourse from getting a lot of "How do I docker" issues and e-mails from their clueless users.
So, would you prefer that part just says "installing Docker is out of scope" or should the Discourse developers go through every distro and cloud system and document the specific instructions for that? To do that would completely defeat the purpose of even using Docker at all.
I concur - to elaborate, I'm not actually the one installing Discourse. I support a 'Kumbaya' group of data scientists who have never heard of docker.
Quick directions like these aren't questioned by users who just want to get things done, and they invite security risks just as the parent & article suggest.
I understand that curl pipe sh could have security problems but I also don't see it as that much different than the "normal" and "ok" way of doing things. I would consider something like the below pretty normal.
wget https://whatever.io/latest.tgz
tar xzf latest.tgz
cd whatever-stable
./configure && make
sudo make install
Because of familiarity, we aren't going to be too worried about what we are doing. If we are on a secure system (like a bank or something) then we've probably already gone through a bunch of hoops (source check, research) and we mitigate it like anything else.
What is so different about
curl https://whatever.io/installer.sh | sudo bash
We didn't check the md5s in the first example, so yolo, we don't care about the content of the tarball we just `make install`-ed. We're assuming the webserver isn't compromised and that https is protecting the transfer. Is it because the tarball hit the disk first? Does that give us a warm fuzzy? Is it because "anything could be in installer.sh!!!?! aaaaah!". Well, anything could be in Makefile too right? Anything could be in main.c or whatever.
I agree that curl sh | sudo bash makes my spidey sense tingle. But if I really cared, I would read the source and do all the normal stuff anyway. So I think it's some kind of weird familiarity phase we're all in.
I wouldn't consider what you presented as the "normal" or "ok" way of doing things either, especially not on anything resembling a live (i.e. not development/sandbox) environment.
A distro (or official vendor, or possibly a trusted third-party) repo of pre-built, signed packages would always be my first choice.
If one of those isn't available, my next step would be to create a package for the tool in question, part of which is setting up a file for `uscan` to download new source archives, and compare against the signatures.
In this scenario we (as in the organisation) are now responsible for actually building and maintaining the package, but we can still be assured that it's built from the original sources, we can still install it on production (and even dev, staging, whatever) servers with a simple call to apt/aptitude, and dependencies, removal, upgrades, etc are still handled cleanly.
About "ok". You're right. I probably used a loaded word without context. I too use whatever default package repo, followed by "extras" or whatever is available. You described a sane and nice process. I guess my point is, at some point we are are assuming "many eyes" (the binaries might be built with the previously mentioned make;configure steps) unless you are auditing all sources which is unlikely. Especially unlikely on dev machines. Even after that it seems like there is an infinite continuum of paranoia.
I find it interesting that binary packages have existed for decades and yet `rpm etc` knowledge is rare. Why did curl sh become popular? Why doesn't every project have rpm|deb download links for every distro version? Why don't github projects have binary auto-builds hosted by github? I'd argue that it's too difficult. Binary packaging didn't succeed universally. For deployment, containers are (in the end) easier.
But the original article is conflating container concepts and user behavior (not wrongly). If docker hub does end up hosting malware-laden images, it would be interesting emergent behavior but it would be orthogonal to containers. Like toolbars. Toolbars probably aren't evil. A vector for evil maybe?
> I find it interesting that binary packages have existed for decades and yet `rpm etc` knowledge is rare
What makes you think the knowledge is rare? Among developers who actively target linux distributions I would imagine the opposite is true.
Even a number of the referenced curl|bash offenders are just using that as a "shortcut" to add their own apt/yum repos and calling apt-get/yum to install their binary package(s).
Because now your ability to install your mission-critical software is dependant upon https://whatever.io actually being up. Which it certainly won't be forever.
Or, you know, maybe someone updated the whatever.io installer to make it 'better'. But you are trying to debug some problem and you made one image last month and another one this month and you're pulling your hair out trying to figure out why they are different. Oh, it's because some text changed on some web site somewhere.
You've taken a mandatory step and put it outside your sphere of control.
Good point. I guess you could still wget the script though. It's maybe like ./configure over http? I guess even if you could do it, it's probably not culture. A Dockerfile would probably just curl sh the thing and not wget it. So the default culture probably does depend on whatever.io being up.
Outside of a development environment, you'd run that ./configure && make install step on a build slave that creates a nice RPM or Debian package of it for you which you can install without fear that the build scripts install backdoors, download obsolete software or wipe the filesystem.
With a good build system (eg. autotools) writing an RPM spec takes almost no time at all and if you have the proper infrastructure in place for building packages, you can have something workable in a very short time.
Self-packaged RPMs also don't need to be quite as high-quality as ones you might want to include in a distribution, so if it makes sense for your use case, it's perfectly okay have "bloat" (eg. an entire python virtualenv) in your package.
It's bad because `sh`, `bash`, etc. don't wait for the script that's being piped into it to finish downloading before it starts executing it. So, for example, if you're running a script with something like
# remove the old version of our data
sudo rm -rf /usr/local/share/some_data_folder
and the network connection cuts out for whatever reason in the middle of that statement (maybe you're on a bit of a spotty wireless network), the resulting partial command will still be run. If it were to cut off at `sudo rm -rf /usr`, then your system is in all likelihood going to be hosed.
* Using checkinstall to create a local deb/rpm which can be easily installed/removed later instead of "make install".
* What if installer.sh says "rm -rf /tmp/PACKAGE-build" and the connection is interrupted just after the first "/", you now have "rm -rf /". Oops.
* configure will tell you what files it needs, and apt-file will tell you want dependencies to install.
* I know what make install does. I know make. Who wrote installer.sh? Do they know anything about writing good software? Steam wiped out home directories, who knows what these people do
I think part of this is because there aren't any trusted, fully open source, artifact repositories that work with the various package indices out there.
Like, most of the way deployment should work is that you come up with some collection of packages that need to be installed and you iterate through and install them. Bob's your uncle.
Thing is, all the packages you need live out in the wild internet. Ideally, you'd just be able to take a package, vet it, and put it in your local artifact store and then when your production deployment system (using apt or yum or pip or gems or maven or whatever) needs a package, it looks at your local artifact store and grabs it and goes about its business. Never knowing or touching the outside world.
And your developers would all write their apps to deploy through the normal packaging methods that everyone and their mother is already familiar with and they could just put them into the existing package index as well.
But you've gotta lay out pretty serious moola (from when I last looked into available solutions to this) or set up a half dozen different artifact stores if you want to do things that way. And good luck managing your cached and private artifacts if you do. And on top of that developers don't necessarily know how to set up a PyPi or a RPM index or whatever so that the storage is reliable and you've got the right security settings or whatever else. (I know I sure don't and I'm not really interested in reading all of the ones I'd end up needing).
"And on top of that developers don't necessarily know how to set up a PyPi or a RPM index or whatever so that the storage is reliable and you've got the right security settings or whatever else. (I know I sure don't and I'm not really "
Setting up RPM is shockingly easy. It can get more complex, but the basic system is:
REPOBASE=/srv/www/htdocs/
createrepo -v $REPOBASE
gpg -a --detach-sign --default-key "Sign Repo" $REPOBASE/repodata/repomd.xml
gpg -a --export "Sign Repo" > $REPOBASE/repodata/repomd.xml.key
That will create a repo from all the .rpm files in the REPOBASE. Also you will of course need a GPG key pair, but that can be generated with `gpg --gen-key` where you give it a description of "Sign Repo" (or change the above commands to the key description you used).
Then you get to decide on the deployment machine if you want to trust the repo (or you don't trust any and import the key via some other process, aka direct gpg import).
Of course you can find a bunch of more detailed explanations with $SEARCHENGINE, but if it takes more than a day to figure it out, your doing something wrong.
Building a set of RPM's isn't that much harder if you have a proper build system. But these are the kinds of things you give up when you decide to grab the latest immature hotness created by someone on their day off.
With docker, as referenced in TFA... you can simply vet a base image, and use that for your application... upgrades? create a new/updated base image and test/deploy against that.
Same as everything: look at how it was built. Many of the images are built by CI systems according to Dockerfiles and scripts maintained in public GitHub repos. Audit those, then use them yourself if you're worried about the integrity of the services and systems between the code and the repository.
It's easy enough to download a given/checked version of the script at http://foo.com/ubuntu/install and have that copied and run inside your docker image... for that matter, it's usually adding a given repository to your repo manager, then installing a given package from that software's corporate sponsors.
I don't think the problem is as rampant as it's made out to be in TFA... that said, most people don't look at said script(s), so it's entirely possible something could have been slipped in. For that matter, I think the issues outlined in the article relate more to overly complicated Java solutions (the same happens in the .Net space) that are the result of throwing dozens of developers some with more or less experience than others at a project, and letting a lot of code that isn't very well integrated slide through whatever review process does or doesn't exist.
Obviously a config / deployment system, like any other system, will start small and simple and "save a lot of time" but after an infinity of features are bolted on, it'll be infinitely worse than just using a bash script. Even worse, you probably figure out your deployment by hand on one system using bash, then need to translate what worked on a command line into crypto-wanna-be-bash config system (probably creating numerous bugs in the translation) then using wanna-be-bash to slowly poorly imitate what you'd get if you just used bash directly...
The last straw for me was trying to integrate some freebsd servers and /usr/ports had like six versions of cfengine none of which worked perfectly with the three versions on the legacy linux boxes. Screw all that, instead of translating bash command line operations into psuedo-bash I'll just use bash directly. IT is an eternally rotating wheel and the baroque inner platform deployment framework has had its day... and being an eternally rotating wheel it'll have its day again in a couple years. Just not now.
Not throwing the baby out with the bathwater, a strict directory structure, and modularity and library approach to error handling and reporting and logging which you can steal from the deploy systems is a perfectly good idea.
Unix philosophy of small perfect tools means I'm using git instead of my own versioning/branching system, and using ssh to shove files around rather than implementing and static linking in my own crypto and SSL system.
I agree with you in principle, but in practice shell scripts are really not the best tool for this sort of job: they tend to be write-only (in the sense that they can be difficult to read months or years later) and can become very hairy and difficult to maintain.
I'd prefer something like scsh (or a Common Lisp or elisp version thereof) for this sort of work: access to a full-fledged programming language and easy access to the Unix environment.
"can become very hairy and difficult to maintain."
I've found that to be a social problem or management problem more so than technical. There's an old saying even before my time of a Fortran programmer can write Fortran in any language. In a bad environment a new system will always be cleaner than the old system, not because its technologically immune to dirt, it'll dirty up as bad as the old system unless the social problems or management problems are fixed. You really can write read only Puppet scripts. Or you can write readable bash. Or even Perl.
Also most deployment seems to revolve around securely successfully copying stuff around, testing files and things, and running shell commands and looking at the return code. Shells are pretty good at running shell commands like those in a maintainable easily readable and troubleshootable fashion. Its possible that a deployment situation that more closely resembles a clojure koan than the previous, might have some severely blurred lines. And there's always the issue of minimizing the impedance bump between the automated deployer and the dude writing it (probably running commands in a shell window) and the dude troubleshooting it at 2am (by looking at the deployment system in one window and running commands in a shell window next to it to isolate the problem). I would agree that cleaner library/subroutine type stuff in shell would be nice.
And you are correct, scsh is really cool but two jobs later some random dude on pager duty at 2am is more likely to know bash or tcsh. Principle of least surprise. I suppose if only scsh guys are ever hired... Then again as per above most deployment is just lots of moving stuff around and running things so its pretty self explanatory. But if the work is trivial, don't deploy a howitzer to swat a fly.
Maybe another way to look at it is if you're doing something confusing or broken, plain common language will clear things up faster and more accurately than using an ever more esoteric domain specific language. Or some folk saying like "always use the overall simplest possible solution to a complex problem".
There is the "don't reinvent the wheel" argument. I have a really good network wide logging system, a really good ssh key system for secure transfer of files, a strong distributed version control system to store branches and versions, a strong SSL infrastructure, a stable execution environment where upgrading bash probably won't kill all my scripts, a strong scheduled execution system... I don't need a tight monolithic collection of "not so good" reimplementation of the above, running that is more painful that rolling my own glue between the strong systems I already have. And using the monolith doesn't mean I get to abandon or ignore the "real" strong infrastructure, so the only thing worse than running one logging/reporting system is having to admin two, a real enterprise grade one and a deployment-only wanna be system. I did the puppet thing for many years. So sick and tired of that.
Thank You for the Wikipedia link - I was looking for the name of the "thing" people are doing when they write all those WebGL JavaScript frameworks and such. Now I know that they are creating poor replicas of things that normally run on the desktop itself.
What do you expect them to do, download .tar.gz, extra, read every line of code and them make; make install?
Or just make; make install? How is that any different?
You can usually get PGP signed hashes for tarballs distributed by serious entities. If someone is distributing software and provides no way to check that it is genuine, you shouldn't run it...
Go look up how you install snort or bro on centos. You have to either install from source, or install from a rpm from there website which may or may not have issues. This means you lose dependency management, and update management. Pure madness
I've decided that unless you're ok with running a very restricted set of ancient applications, don't even try to use CentOS. I've seen multiple billion dollar companies who can't seen to avoid f'ing up the yum repos on CentOS.
I'm not able to go full docker on my machines @work, but I do have some statically linked tarballs. There is a reason apps that deploy in hostile environments (skype, chrome, firefox) bundle most of their dependencies.
There's nothing wrong with curl | sudo bash style setups as long as it's over https and the certificate gets checked.
The advantages are that it's easy and you can make it work on almost all unix-like systems out there.
The only disadvantage is that you have one additional weak point: The server can get contaminated. Before you had to contaminate one of the many developer machines / build machines.
The situation hasn't been better before. Install media always got downloaded without ssl encryption or any certificate checks. This is still the same, but at least you won't get a hacked kernel today if you use secure boot.
Just because it's easy to run doesn't mean it's easy to support or maintain. Chances are `curl | bash` scripts aren't designed for your particular OS, so it's yet another form of software that you have to learn how to update, as opposed to using the OS-level update mechanism, such as yum, apt, or even brew to some extent. Being a good sysadmin doesn't stop at installing the software. Most of the hard (boring) work is in maintaining systems and keeping them updated and secure. Blind install scripts make this job impossible.
There is a very big difference between installing something on your dev machine to just get it started and deploying something into production. `curl | bash` is okay for setting something up on a dev machine where the only one that needs to use it is you. For productions machines, it's completely inappropriate[1].
[1] This is somewhat mitigated by things like Docker, but I'd still argue that you don't want to have an ephemeral installation method for containers either. You should have fixed versions that are installed by either a package manager or at least a Makefile.
Not to mention that in plenty of environments production systems don't have access to the internet to begin with, so curl/wget | bash is a non-starter.
There's nothing wrong with curl | sudo bash style setups as long as it's over https and the certificate gets checked.
Even assuming the URL's publisher is trustworthy (which is a poor assumption to make, ever), you forget SSL / HTTPS is broken, that the NSA has established MITM on the entire internet, that your installation process (which should be both versioned and repeatable) now has zero versioning and all the entropy of the network plus bonus entropy.
I'm guilty of using this method in my side project (https://github.com/grn/bash-ctx). My goal was to solve the installation problem quickly. I absolutely would love to offer proper installation methods. However my experience with building *.deb packages makes me think that it's not something that I'd like to do (especially it's a side project).
The question, therefore, is: what is the simplest alternative installation method for OS X and Linux?
There is some truth in this, yes. On the other hand, maven (mentioned by the author) clearly was very successful at abstracting from the tools we use. I can remember how much time I wasted with build scripts and dependency management and all that before. (And I still do on some other platforms.) The problems only arise if the abstraction is not working well enough. This might indeed be true for containers - there is maybe too much complexity in there that currently can't be properly encapsulated.
The obvious question is: what's the real problem with that?
A container is a container, as long as docker itself has not bug, the container can only harm the containers content.
Most problems exists in the custom created software in the container (e.g. web-services with bugs, backdoors, ....), this will be a problem for Docker, VMs, Real-Servers, whatever too.
The real problem is the interoperability of different container, if you link the whole data, without any audit, to another container, you can have a problem, but this problem is not docker specific.
>> A container is a container, as long as docker itself has not bug, the container can only harm the containers content.
Presumably a container has network access of some sort? Malicious code could start probing and attacking anything exposed that way.
>> this will be a problem for Docker, VMs, Real-Servers, whatever too.
The implication is that you wouldn't get into this situation with a 'Real-Server' so easily, because you wouldn't just download an image and run it, without having an update/patch strategy or having much more idea of what's going on inside it.
But you assume that a container HAS full network access.
A firewall must be configured, but a firewall must be configured for a VM too.
My point is, that their is not so a huge difference for production systems.
>> But you assume that a container HAS full network access.
No, I'm presuming it has some sort of network access, a malicious container could (for instance) still probe other containers for vulnerabilities, serve malware etc etc without full network access.
>> A firewall must be configured, but a firewall must be configured for a VM too. My point is, that their is not so a huge difference for production systems.
If you're downloading VM images from somewhere and running them without checking what's in them you'll run into the same problem, sure.
The problem being pointed out here is that when applications are bundled outside of the purview of a packager like debian you -
- don't have as much trust in the origin of the app
- don't have an easy way to keep up on library patchlevels etc for security
Not true if the software in your VM or RM is managed by a package manager and comes from a place that issues security updates, patches etc.
One of the criticisms in the article is that much of what's going on now, either with containerisation or weird build systems like Hadoop's, misses out on this.
"Maven, ivy and sbt are the go-to tools for having your system download unsigned binary data from the internet and run it on your computer."
You should setup a maven repository (Nexus, Artifactory) for your organisation if you want to have more control on binaries.
Seems that artifactory can host docker files: https://www.jfrog.com/confluence/display/RTF/Docker+Reposito...
Right, do folks really belief Maven, Ivy, Gradle, Sbt are tools you use in production? These are developer tools for use on workstations and CI servers. If you want to promote your stuff to other environments like production use your own private repository (Nexus, etc).
They may be if you have a team without much sysadmin experience. The way you develop could be the way you deploy to production.
These are the same teams that have overprivileged accounts for the database or sudo-enabled users running applications or chmod 777 all over the place.
Even things like Chef cookbooks have this going on. If you want to build from source because it's not in your repository, then you're necessarily going to need to drag in sbt or gradle. (see https://github.com/hw-cookbooks/kafka/blob/develop/recipes/d... as an example). Sure you could figure out the mirrors and download the correct binary from the website. You could also use this recipe to compile everything and then package it up to host yourself. (Both of these actions require writing custom recipes). Not everyone has time to do this, and this magical recipe you found online works great on the development server! Just add it to the production server and now we've just used sbt in production on a software team.
make is the least-auditable build tool imaginable. You don't have to obfuscate a Makefile, they come pre-obfuscated; you could put the "own me" commands right there in "plain" Make. Not to mention that it's often easier to tell whether a Java .class file is doing anything nefarious than whether a .c file is. How many sysadmins read the entire source of everything they install anyway?
Maven, on the contrary, is the biggest single source of signed packages around. Every package in maven central has a GPG signature - the exact same gold standard that Debian follows. The problems Debian faces with packaging Hadoop are largely of their own making; Debian was happy to integrate Perl/CPAN into apt, but somehow refuses to do the same with any other language.
> Instead of writing clean, modular architecture, everything these days morphs into a huge mess of interlocked dependencies. Last I checked, the Hadoop classpath was already over 100 jars. I bet it is now 150
That's exactly what clean modular architecture means. Small jars that do one thing well. They're all signed.
Bigtop is indeed terrible for security, but its target audience is people who want a one-stop build solution - not the kind of people who want to build everything themselves and carefully audit it. If you are someone who cares about security, the hadoop jars are right there with pgp signatures in the maven central repository, and the source is there if you want to build it.
Makefiles don't really enter into it and getting software signed by the developer isn't that valuable or useful.
The value of debian is not that they package (or repackage) everything into deb files but that they resolve versioning and dependancy conflicts, slip security fixes into old versions of libraries (when newer version break API/ABI), and make it possible to integrate completely disparate software into a system. They also have a great track record at it.
Maven does not do any of these things; Maven does nothing to protect the system administrator from a stupid developer, it just makes it easier for their code to breed and fester.
You must understand that the sysadmin has an enormous responsibility that is difficult for programmers to fully appreciate: You don't feel responsible for your bugs, you don't feel responsible for mistakes made by the developer of a library you use, and you certainly don't feel responsible for the behaviour of some other program on the same machine as your software, after all: Your program is sufficiently modular and scalable and even if it isn't, programming is hard, and every software has bugs.
But the sysadmin does feel responsible. He is responsible for the decisions you make, so if you seem to be making decisions that help him (like making it easy for you to get your software into debian) then he finds it easier to trust you. If you make him play whackamole with dependencies, and require a server (or a container) all to yourself, and don't document how to deal with your logfiles (or even where they show up), how or when you will communicate with remote hosts, how much bandwidth you'll use, and so on: That's what Maven is. It's a surprise box that encourages shotgun debugging and using ausearch features to do upgrades. Maven is a programmer-decision that causes a lot of sysadmins grief a few months to a few years after deployment, so it shouldn't surprise you to find that the seasoned sysadmin is hostile to it.
> The value of debian is not that they package (or repackage) everything into deb files but that they resolve versioning and dependancy conflicts, slip security fixes into old versions of libraries (when newer version break API/ABI), and make it possible to integrate completely disparate software into a system.
Maven has exactly the same capabilities as deb does - you can depend on versions, depend on a range of possible versions, exclude things that conflict and so forth. And it puts even more emphasis on fully reproducible builds (with the aid of the JVM) - in that respect it's closer to nix than apt.
> But the sysadmin does feel responsible. He is responsible for the decisions you make, so if you seem to be making decisions that help him (like making it easy for you to get your software into debian) then he finds it easier to trust you. If you make him play whackamole with dependencies, and require a server (or a container) all to yourself, and don't document how to deal with your logfiles (or even where they show up)
Wow, self-important much? Too many sysadmins seem to forget that the system exists to run the programs, not the other way around.
> If you make him play whackamole with dependencies, and require a server (or a container) all to yourself, and don't document how to deal with your logfiles (or even where they show up), how or when you will communicate with remote hosts, how much bandwidth you'll use, and so on
On the contrary, maven makes the requirements much simpler. I have literally one dependency, a JVM, so it can run on any host you like (no need to worry about shared library conflicts with some other application). It needs to download one file (shaded jar) from our internal repo, and execute one command to run it. That's it.
> That's what Maven is. It's a surprise box that encourages shotgun debugging and using ausearch features to do upgrades.
No, it's just the opposite. All the dependencies and project structure are right there in declarative XML. It's what make should have been.
> No, it's just the opposite. All the dependencies and project structure are right there in declarative XML. It's what make should have been.
When make was written most machines would have just exploded at the sight of a typical build.xml, and downloading tens or hundreds of packages from anywhere was simply out of the question.
Also, 'dependency' means something completeley different in make as opposed to maven - I don't think modern build systems do even care much for make-style deps.
> When make was written most machines would have just exploded at the sight of a typical build.xml, and downloading tens or hundreds of packages from anywhere was simply out of the question.
Sure. But the notion of doing things declaratively existed (Prolog predates make by five years). And the biggest difference between make and the scripts that preceded it is that it's more structured, with a graph of targets rather than just a list of commands.
If you add the ability to reuse libraries of targets (something that sort-of exists via implicit make rules), restrict targets to something a little more structured than random shell commands, and - yes - add the ability to fetch dependencies (including target definitions) from a repository, you end up with something very like maven.
You really nailed it. Among my duties are systems administration for a company that works with a lot of software development vendors. We have a user acceptance team that makes sure that we get what we ordered, that the QC stays at a high level. So functional problems, that's their deal. But they're not sysadmins, they can't easily see what developer choices make administering the servers more complicated, more fragile, more expensive, or more insecure. This shifts my job from the end of the process (here, run this!) to the beginning (hey guys, let's use these tools instead, it'll make everyone's lives easier).
As such I'm very pro containers as they will eliminate a ton of deployment effort and allow me to manage different environments much more easily. But it means that there needs to be a much bigger magnifying glass on the container contents early in the process as opposed to the moment of deployment.
Debian has a terrible track record. Just look at the OpenSSL/Valgrind disaster. As a former upstream developer myself (on the Wine project), all Linux distros found unique ways to mangle and break our software but Debian and derived distros were by far the worst. We simply refused to do tech support for users who had installed Wine from their distribution the level of brokenness was so high.
You may feel that developers are some kind of loose cannons who don't care about quality and Debian is some kind of gold standard. From the other side of the fence, we do care about the quality of our software and Debian is a disaster zone in which people without sufficient competence routinely patch packages and break them. I specifically ask people not to package my software these days to avoid being sucked back into that world.
As a sysadmin you shouldn't even be running Maven. It's a build tool. The moment you're running it you're being a developer, not a sysadmin. If there are bugs or deficiencies in the software you're trying to run go talk to upstream and get them fixed, don't blame the build tool for not being Debian enough.
I find it weird that you consider 'packaging' to be something a sysadmin should do, but 'building' to be something they should not do. Aren't they both forms of 'prepping code for use'?
And then state that you don't want your own software packaged. So, if a sysadmin is not allowed to build and not allowed to package, how are they supposed to get your code into production? "curl foo | sh"?
I read that the other way around. He specifically asks people to refrain from packaging his software. I think the implicit implication is that he does want sysadmins to run his carefully constructed build scripts in order to install the application.
I don't consider packaging to be a sysadmin task. On any sane OS (i.e. anything not Linux/BSD), packaging is done by the upstream developers. That doesn't happen on Linux because of the culture of unstable APIs and general inconsistencies between distributions, but for my current app, I am providing DEBs and woe betide the distro developer who thinks it's a good idea to repackage things themselves ...
Well, we're going to have to agree to disagree there, because I think Windows packaging is fucking insane.
One of the things I loved about my move to linux and .deb land was that if I uninstalled something, I knew it was uninstalled. I didn't have to rely on the packager remembering to remove all their bits, or even remembering to include an uninstall option at all. Or rely on them not to do drive-by installs (which big names like Adobe still do, out in the open). And not have every significant program install it's own "phone home" mechanism to check for updates. The crapstorm that is Windows packaging is a fantastic example of a place where developers love and care for their own product, but care not a jot for how the system as a whole should go together.
Debian feels like a distribution maintained by a bunch of sysadmins: People who have shit to do, and who understand that the purpose of a machine is to get stuff done, not to run some software.
A lot of sysadmins believe since they are responsible for the software, they need to be able to build stuff and fix some stuff themselves (i.e. it can't wait for upstream). In my experience, it's usually something stupid (like commenting out some logspam), but it's critical enough that I can imagine a lot of shops making it mandatory to ensure they can do this.
Really proactive sysadmins do try to run fuzzers and valgrind and do try to look for bugs rather than waiting for them to strike. And sometimes they get it completely wrong, as in the OpenSSL/Valgrind disaster, but they usually ask first[1].
Now I don't agree with everything Debian do, and I don't want to defend everything they do, either, but I think programmers in general need to get out a certain amount of humility when dealing with sysadmins: Because when these sysadmins say that they're not going to package hadoop because the hadoop build process is bullshit, it isn't appropriate to reply "well you guys fucked up openssl, so what do you know?"
One thing that would help is if we didn't look at it as Programmers on one side of the fence and Sysadmins on another side. Programmers have problems to solve, and sysadmins have problems to solve, and maybe you can help each other help solve each other's problems.
> The value of debian is not that they package (or repackage) everything into deb files but that they resolve versioning and dependancy conflicts, slip security fixes into old versions of libraries (when newer version break API/ABI), and make it possible to integrate completely disparate software into a system. They also have a great track record at it.
The analog of the aspects of what Debian does that you're talking about here in the HStack world are companies like Cloudera, who, surprise, make their stuff available as debs and PPAs.
Building your own Hadoop from source and complaining that the resulting product is unvetted is sort of like doing a git pull of all the Linux dependencies and building that.
And yes, if I'm using Debian and didn't add any PPA or extra sources, then the Debian Foundation IS the sole trusted source of software. And you do that because you know hey won't fuck up the system, which (and that's the whole point of this thread) the others certainly don't.
Now Debian is telling you: we see now way to distribute this software and guarantee what you are getting or that it won't fuck up the system.
So everyone who runs Hadoop is installing junk? Seems like plenty of other companies have been able to build businesses on it without adhering to your Debian-only rules...
I could sign anything I like, but that doesn't make it any more secure for you to curl it into /bin/bash.
Signatures are about who signs it, and that's not something mvn has solved at all. Mvn is a free-for-all of binary code that very well could own my system, wheras debian is a curated collection of software which the debian maintainers have signed as being compiled by their systems with no malign influence and having met at least some bar.
I'd trust foo.jar signed by debian over foo.jar signed by bobTheJavaBuilder@gmail.com anyday... and mvn only gives you the latter.
So yeah, sure, they're signed, but it doesn't actually matter if you don't take the time to hook into the chain of trust (and believe me, mvn does not ask you to trust your transitive jar dependency 50 down the line) or have a trusted third party (debian) do their own validations.
> wheras debian is a curated collection of software which the debian maintainers have signed as being compiled by their systems with no malign influence and having met at least some bar.
And not only that, by shipping the source and requiring that binaries can be built from the source, who signs it is no longer blind trust. Others can audit it.
Reproducible builds should improve this even further.
> wheras debian is a curated collection of software which the debian maintainers have signed as being compiled by their systems with no malign influence and having met at least some bar.
This comes with a huge tradeoff, and I guess it's that tradeoff that makes developers like myself opt to sometimes even pipe the cURL to bash. I almost never download any software I actually plan to use through official system repositories, because whatever comes out of apt-get, it's almost always two years behind the last release and missing half the features I need. Sure, I'll apt-get install that libfoo-dev dependency, because I don't care what version it is as long as it's from the last decade. But for any application I actually need to use, it's either git repo or official binary download.
That's because that those of us who actually run real production systems with real production traffic with real production users who pay real money aren't stupid enough to run a bleeding edge code since we see how many "ooops" bugs are listed in the CHANGELOG
> whatever comes out of apt-get, it's almost always two years behind the last release and missing half the features I need
As a sysadmin, I love that, but I've had to come to terms with the fact that some developers have the attention span of hummingbirds who had Cap'n Crunch for breakfast ("two years old" is still very new software from an administration perspective).
So, I've basically accepted the fact that whatever stack the developers use I'll build and maintain directly from upstream -- with the price being that the version and extensions/whatever used are frozen from the moment a given project starts.
Who can't read a Makefile? Who can't at least read the output of make -n? It's terrifying to me that you're suggesting that people can't and don't.
It's not even a security thing. I've had poorly-written Makefiles that would have blown things away thanks to an unset variable on a certain platform, for example.
> Who can't read a Makefile? Who can't at least read the output of make -n? It's terrifying to me that you're suggesting that people can't and don't.
Can I read a Makefile? Sure. But 90%+ of Makefiles these days are 12000 line automatically generated monstrosities. It's not worth my time to bother opening the Makefile in a text editor in case it isn't, and I'd be amazed if many people did.
make -n you can do I guess. But unless you're also auditing all the source code I'm not sure there's a lot of value in it.
> It's not even a security thing. I've had poorly-written Makefiles that would have blown things away thanks to an unset variable on a certain platform, for example.
> As far as I know, it's also still standard practice in
> most companies to either read the source code of open-
> source stuff before deploying it to production (binary
> or build) or get a support contract from someone else
> who has
I'm afraid I have no better, more cogent response for this than 'lol'.
At the risk of my karma I'll have to maintain that for companies who are subject to regulation (publicly-traded companies, banks, etc.) what I said is still standard. Unless you have any specific instances to the contrary you're willing to offer?
Coming from my background (DoD, don't laugh.), code review of anything other than in-house developed applications never occurs. In any instance open source is used, it is mandated that it come with a support contract (per DISA STIG) which provides the support and accountability the organization is looking for.
So, with regards to your assertion, the second clause (support contract)? Definitely. the first (code review), never.
That's the view from my side of the fence, anyways.
In defense, it's too expensive to do in house. However, security software and some operating systems, either closed source or open, are evaluated under Common Criteria (https://www.commoncriteriaportal.org/) But the evaluation process is rather long and often lags behind the current version by a year or more in some cases.
I'll queue this to read later, but just reading the executive statement, it seems to jive with a lot of the discussion here.
The issue isn't that code review /shouldn't/ be taking place, or even that there aren't directives stating that it should be done. It's that it isn't being done, and that's a problem.
I'd love a specific example? Because in many countries if they're publicly-traded or subject to other regulations (such as Basel, etc.) any company that didn't would be breaking the law.
I'd love a specific counter-example. I've never in my life encountered a sysadmin who read all or even part of the source code of any major package before deploying it. I mean, do you read the source code of the Linux kernel, PostgreSQL, nginx, OpenSSL, ... before installing it?
If Basel requires that level of auditing, I'd love to see a cite. There are some pretty heavy-handed regulations out there, but this would be clearly unworkable.
Given the context, it's probably a bad idea to name employers, but I've literally never seen this happen at any company I worked for, and I did consulting, so that's quite a few.
I wonder what laws you're referring to that requires companies to have read the innards of products they deploy.
Even the "getting a support contract from someone who does". Do you think for example Red Hat has read every single line of code shipped in RHEL?
> As far as I know, it's also still standard practice in most companies to either read the source code of open-source stuff before deploying it to production (binary or build) or get a support contract from someone else who has.
Reading the source code might be the case if you are in a large enterprise that can afford to keep those programmers busy or needs to actually vet the code but I honestly doubt it. Just check the loc count on something as common as Hadoop (close to 1 million [1]) and that is excluding the "stack" described in the article. I wanna bet you, because we all know how volatile the job market is that even the guys building or supporting the software did not write or read even half of that code.
I suspect this is why when many companies decide they want to use open source stuff they contract with companies like GitHub, who absorb the risk of using git, etc, rather than trying to vet the code themselves. I still have a hard time envisioning anyone in any position of authority in any credible company accepting the idea of installing open source software without vetting it one way or the other. If the shit did hit the fan their career would be pretty much over.
companies like GitHub, who absorb the risk of using git
Except they don't do that at all?
If a bug in git would cost your company a lot of money, why do you think GitHub would be accountable if you have some enterprise deal there?
I still have a hard time envisioning anyone in any position of authority in any credible company accepting the idea of installing open source software without vetting it one way or the other.
Well, that is an entirely different thing. People TEST stuff before deploying it. But that has little to do with code reading or mandatory support contracts.
Excellence in (open source or otherwise) security practices are the exception, not the norm.
I've worked in multiple large companies and even getting package signing turned on requires a lead pipe. Docker and tools can enable an org to move those types of responsibilities "over/down" to the developer as well, so that now there is no neck-beard encrusted gate at all.
I've never seen anyone evaluating code in the environments I've been in. In many ways, I run into the reverse, where it's a given that these build processes are obtuse, and so they're simply untrusted as opposed to pulling them apart. Administrators either rely on paid products, or long, manual processes.
I've unfortunately had more than one mind numbing conversation where another administrator will tell me that it's a "script" and therefor, I don't know what it does, and that there "may be" some hidden black magic that will instantly pwn all of our systems. Attempting to explain that I verified the functionality line-by-line brings blank stares, as if it's utterly impossible for someone to derive the dark magics that are code.
Thankfully, I haven't run into many instances where software is blindly installed as the author relates, but his point in that "no one knows how it works" combined with the lack of attribution creates an environment of mistrust which greatly limits our ability to take advantage of open source software.
Not even close. Docker now has some terribly attempts at signing images on their registry iirc (docker inc signs them for the docker client).
There is no option for me, as a user, to build and sign my own image with my own pgp key afaik.
My organization might already have a chain of trust, and docker is asking me to ignore that and just trust their signatures (which also only work on dockerhub as of docker 1.5... don't know about 1.6 because you can't use docker for at least a month after a release else security holes galore).
Docker did nothing to encourage signing containers. At 1.0 they had no capability to do any signature, verification, whatsoever. It's being added as an afterthought, and poorly.
If you look at the AppContainer specs, signatures (pgp based) were built in from the very beginning, it lets me create my own chain of trust (including incorporating other's keys), sign my own images, trust someone elses signature, does not trust the transport or storage medium, and has integration with the clients.
If you want to convince me docker cares, you're going to have to give me examples of where they didn't fuck up...
Tell me how I can use docker's tools to sign my own images, optionally trust my friend Alice, and securely download images that she uploaded to her own registry or dockerhub but signed with her gpg key without me having to trust docker inc.
To my knowledge, all docker has right now is doing a 'tarsum' of images which assumes the registry is trusted and, even given that, can be downgraded for backwards compatibility reasons fairly trivially.
I agree that security and provenance is a real issue in Docker. It is however being worked on, and it will be solved. Presumably we will end up with some sort of app-store like framework with proper signatures and verification.
Docker can't do everything at once. Give them a chance. The new version of the registry is a major step forward in this regard.
In the meantime, what you can do is take redhat's advice. Rather than using a registry to get your images, operate a download site which stores archives of docker images that you can import with `docker load`. You can then also store signatures and check them yourself.
Cool, they hired some people, but I haven't noticed better security for it yet.
Security is a real issue in docker and it is being worked on, but I don't think "give them a chance" is a justifiable response. They're not focusing on it strongly. They already should have focussed on it and didn't. Their entire codebase was written without a security design in place, so there's likely deep-seated refactoring that'll need to be done before any new security-related features should be trusted.
They're working harder on monetizing and pushing docker as a production-ready standard as far as I can tell... I can understand not doing security before functionality, but it absolutely should be there before 1.0 or before you encourage others to use your software.
Docker has already lost any chance of me trusting their security with their lack of focus on it and I don't think it's excusable.
And if I'm doing what you say at the end, why the hell would I be using docker anyways then? I can already turn a tarballed fs into a linux container without docker (ty lxc); I thought the whole point of docker was sharing images and building on them and ... and having massive security flaws. Right.
I agree that security should have been in place before they went 1.0. However, if you look at the work on the version of the registry (docker/distribution on github), they are taking things more seriously and trying to get the basics right.
I find your last point a bit strange. We all know the Docker development experience is a lot better than raw lxc. I'm saying you can (and probably should) be more careful about provenance than the Docker Hub is. Note that there are alternatives to the Hub with better provenance stories e.g: https://access.redhat.com/search/#/container-images (from https://securityblog.redhat.com/2014/12/18/before-you-initia...) This might make things a bit more awkward than it was before, but it's still not the same as raw LXC.
I feel your anger and I think it's understandable, but that doesn't mean things won't get better.
As an ex sysadmin I really like the container infrastructure. Manage the whole configuration on the main machine with puppet and deploy the blackbox applications (everything ruby and java related) with docker/rocket.
It's nice to have the option. Containers are awesome for many things/projects, but sometime you just want to run the damn application on a server of your choice, without any container stuff.
I can't remember what the application was, but I've seen an application where the only installation instructions where for Docker. That's just plain silly.
My concern with containers is that the wrong people will use it. There is a ton of software out there with just barely runs and make all kinds of assumption about it's environment. I fear that rather than design better, more correct software, these people/companies will start packing up their development environments as containers (more or else) and just ship those. Of cause that's no reason to discourage the use of containers, we just need to be critical of what is inside them.
We've been doing some work with Elastic Beanstalk lately, and - while it certainly does one or two things that are extremely clever and useful - in the end it just feels like this bizarre mix of complete magic and incredibly convoluted arcana. Everything feels very out of our control and locks us into an ecosystem that considerably limits our choices and flexibility (unless we invest the time in becoming experts in EB, which really isn't particularly something we have the time for). And, as the author of this post says, the security ramifications, while orthogonal, are also deeply troubling.
I was doing sysadmin the "right way" a long, long time ago, and I don't see much difference. Maybe the author regularly does full audits of the source code of every package he downloads, and of course disassembles every executable and library in the underlying OS, but most of us don't. There's no wisdom or security to be gained from the act of running "make", much less "make install".
> Maybe the author regularly does full audits of the source code of every package he downloads, and of course disassembles every executable and library in the underlying OS, but most of us don't.
What matters is that the source code is auditable. It only takes one person to investigate something suspicious, raise a flag and get it fixed.
This is certainly still true for Debian - not being able to build from source is considered a release blocking bug.
It comes down to trusting two key things. Trusting your initial distribution download (which contains the package signing keys), and trusting the toolchain (in a Reflections on Trusting Trust way).
But the deeper you go, the harder it is for malicious code to reside there. In theory it's possible, but in practice I'd like someone to show me some code somebody could have written into the toolchain a decade ago, without hindsight, which could still exist today.
Whichever way, it's clearly far tougher for a malicious actor to compromise a system by injecting something into a distribution ecosystem than it is to inject a signed-by-unknown-reputation binary-only package into the Maven ecosystem.
This is where reproducible builds [0] come in. We can trust our binaries much more if the same build inputs yield the same build output.
Building on that, could we find a fixed point where the OS builds the exact same bootstrap binaries that were used to bootstrap the OS to begin with? [1] That would give us even more confidence that the binaries we're using are as they should be. Interesting place for experimentation.
>> Maybe the author regularly does full audits of the source code of every package he downloads, and of course disassembles every executable and library in the underlying OS, but most of us don't.
This is not the point being made. Trust is often offloaded, say to the debian people, but it is present in most modern linux systems as a basic part of the setup.
>> There's no wisdom or security to be gained from the act of running "make", much less "make install".
'make' is brought up not because everyone should be running "make" or "make install", but because it's a standard and it is understood by many people. It's brought up in the context of hadoop because the hadoop build system appears to be just so complicated and non-standard, including pulling in untrusted sources from all over the place, that it is near impossible to set it up as a well-audited, standardised package.
Given this, it is likely to be a hive of vulnerabilities, either during the setup phase (if any of the third party servers gets compromised or MITM'd) or during deployment (that java VM it pulled in during setup is never going to get patched).
Maybe I am just only another grey beard grumpy developer, but the new generations that grew up with GNU/Linux instead of UNIX, bash the security of other OSes and then go running such commands all the time.
docker and any user in the group docker, or lets say, any user capable of sending commands to the docker daemon running as root - is root on that system.
Every once in a while someone figures out that we could entirely solve the dependency problem by packaging all the dependencies with the application. Everyone gets excited. After a while everyone gets unexcited when the problems associated with this approach become obvious.
Docker is merely a more extreme example of the "package everything with the application" idea...
I'll bite. What's obviously the problem about it? Or, "if it's good enough for Google...."
Vendoring dependencies and static linking is quite popular in executables, not just docker. Dynamic linking and shared libraries seem to be becoming a relic, deservedly.
BTW, the extreme example of "package everything with the app" is the unikernel movement.
This rant is about containers, prebuilt VMs, and the incredible mess they cause because their concept lacks notions of "trust" and "upgrades".
Prebuild VMs? Sure, I wouldn't touch them except for evaluating a project, and for commercial software you may not have a choice.
But docker containers at least usually provide a dockerfile that describes exactly how a binary image is built. You just clone the source repo, audit the few lines of build commands and then build your own private registry. It's nearly no more trust than following the instructions of README or INSTALL. Just because fools are pulling down pre-build images and running in their datacentre, doesn't mean that's what way you should do it. And the problem with 'old-school' sysadmins is they are often far too quick to reject new practices, citing tired excuses based on misunderstandings of the technologies.
Ever tried to security update a container?
Yeah I have. It's easy if you have already built your 'stack' to scale horizontally (which means you have at least 2 or more of everything in a HA or LB config). You rebuild against a fully patched base-OS container, spin-up, send some test load to it & validate, then bring into service. Repeat for rest of nodes at that tier.
If you are trying to be an old-school sysadmin that expects to console or SSH in and run 'yum upgrade' or 'apt-get upgrade' your containers then you are doing containers wrong...
The old-school sysadmins I know scoff at Docker's idea of 'containers'. Linux containers were already a thing, and don't need an entire copy of an OS ported around with them. To them, containers are a way of enveloping a process to limit it, not a way of distributing packaged software. They may or may not be doing 'docker' right, but they certainly know what 'linux containers' are.
Well I should have qualified it with 'docker containers'. But yeah, those that have been around long enough in Linux container land have all dealt with vserver, openvz, lxc, etc, and all of those carried around this 'entire copy' of an OS, per container (ignoring vserver's vhashify). Docker helps you to spin up N containers running all sorts of applications based on the single master image.
Docker, whether your view is good or bad, brings something more than just another container implementation to the table...
> Linux containers were already a thing, and don't need an entire copy of an OS ported around with them.
Neither do Docker containers. You can build off scratch and put the literal bare minimum you need in it. I've done it a few different times. It's rarely done because the time and effort almost never makes up for the complexity and cost, but if your old-school sysadmins are scoffing it's on them.
Ain't my lawn; I consider myself a mid-range sysadmin. I stepped out of support and into sysadmin land about 4 years ago. But I know some 'from-the-birth-of-linux' guys, who live and breathe this stuff in a way I never will. When I get home from staring at terminals, I want to watch movies and play video games, not swear at something on a breadboard :)
> If you are trying to be an old-school sysadmin that expects to console or SSH in and run 'yum upgrade' or 'apt-get upgrade' your containers then you are doing containers wrong...
A container is a chroot environment for running a service. Basically. It's perfectly possible to 'yum upgrade' or 'apt-get upgrade' them.
I think what you're trying to say is this is a bad idea because containers are not supposed to be managed individually. The magical fluffy dream of Docker is to never have to manage another individual OS again; just make one image, then kick it out to all your machines and make them use the new image. No resolving dependencies, no verifying individual file checksums, no upgrading or downgrading or conflicting different package versions for every server and service. Just do it once, and then push it out everywhere, and everything magically works.
Right?
Here's the thing: Containers don't do away with the idea that a developer might use a totally bleeding-edge piece of software to update one of your many apps, and that you may end up with multiple incompatible versions of software on your systems. In fact, Docker kind of trades on that as a feature. "Install any shit software you want and it'll never conflict with other containers!" But this lie is shown for what it is once you start looking at containers as individual physical machines.
Back in the day we had to 'yum install some-specific-architecture-and-version-of-this-package' on a particular machine to make that machine serve that software. You do the same with Docker, because you have a particular Docker container with a specific version of the package, and all the other packages and OS requirements in that container. You still have a one-off machine to install, maintain and troubleshoot. The only differences are it isn't physical anymore, and you perform updates on the image, not on the machine.
Just like you ended up with a machine (or three machines) with three different versions of BDB, you end up with three containers with three different versions of BDB. Instead of using 'apt' or 'yum' to install them, you write their Dockerfiles and build them, test them, then roll out their updated images. You do a lot more from scratch now because a Linux distro hasn't done the work for you, and so you also run into all the headaches that someone packaging software for a Linux distro usually runs into.
One of the worst things about the 'new devops way' seems to be the non-reproducibility of things like Dockerfiles. To build a Docker image we slap together whatever bleeding-edge files we had on date X-Y-Z, and whatever comes out at the end is considered 'production'. Ignoring the patches, the quality control, the stable released software distributed by distributions on reliable mirrors, and generally without tuning the software at all for the particular system you're running it on. Try to build that Dockerfile again in a year and suddenly it doesn't work the same as it used to, or a bug magically appears on your production system, or you have to apply a patch and now you need to unroll all the commands used for individual stages of the build process for a single package and figure out how to make it still work like the original was built, etc.
Containers by themselves are fine things to use. The problem is how they've effectively sold a lie to everyone that uses them: that there is no sysadmin work to be done. That don't worry, Mr. Javascript Dev, you too can build infrastructure and deploy it without learning the many lessons and best practices of an industry that has been here longer than you've been alive.
This wouldn't even be far from the truth if it was, say, a RedHat-built set of container images, or a Debian-built set of container images. Then at least there'd be an expert who's building software in a reliable uniform way and along a particular standard...
I corrected myself for the other person to mean '[..] upgrade your docker containers'. I heavily use LXC containers (and had used openvz and vserver before that) and treat them as individual servers.
All your points about bad sysadmin practices are OS & container agnostic - they can happen on any platform, don't drag Docker into it. Sure there is a culture of 'docker run somebinaryimage [..]' but those people are the ones that do "curl | sudo bash" as well.
Your claim about non-reproducibility of Dockerfiles is bogus. The result of a Dockerfile build gives you precisely the reproducibility you desire. Every time you run a container from that image built from a Dockerfile, you'll get the same filesystem & environment.
Docker 1.6's "Content Addressable Image Identifiers" addresses your build in a year concern by allowing dockerfiles to refer to a digest to ensure you are building against exactly the image you expect (rather than the result of some build process that yum -y upgrades etc, which I think is what you were getting at).
> Every time you run a container from that image built from a Dockerfile, you'll get the same filesystem & environment
The image (and thus container) are the same. Trying to rebuild it from scratch leaves it not the same, typically because the way people put together Dockerfiles and build images does not follow a standard. And it has to do with the container culture.
FROM centos:centos7
MAINTAINER The CentOS Project <cloud-ops@centos.org>
RUN yum -y update; yum clean all
Right off the bat I think: what the hell? Why are they doing a yum update? A yum update today may very well leave the system in a completely different state than a year ago. There's likely been some package updated in that time, which changes the state of the system. Right off the bat we're screwed.
RUN yum -y install epel-release tar ; yum clean all
RUN yum -y install nginx ; yum clean all
And what the hell is this?! There's no version, no build, no checksum. What the hell did we just install? If those packages change in a year, we're screwed. Not to mention 'epel-release' and 'tar' should be set as dependencies somewhere, and the type of dependencies too.
ADD nginx.conf /etc/nginx/nginx.conf
RUN echo "daemon off;" >> /etc/nginx/nginx.conf
Oh, cool, just use whatever the hell this config is which may or may not be different from the one that shipped with the original package. And let's just modify it for no apparent reason, too. And definitely make sure we have no way to know what version of that file we're using for this image. Lovely.
RUN curl https://git.centos.org/sources/httpd/c7/acf5cccf4afaecf3afeb18c50ae59fd5c6504910 \
| tar -xz -C /usr/share/nginx/html \
--strip-components=1
The hell? We're pulling some random git sources from a git server which i'm willing to bet has no standard mirrors? On closer inspection it doesn't really look like a git server, but a host named git which hosts files whose names are a checksum, though we don't know where this is from or what exactly it refers to. If this server or file disappears, good luck knowing what in hell was being downloaded here.
RUN sed -i -e 's/Apache/nginx/g' -e '/apache_pb.gif/d' \
/usr/share/nginx/html/index.html
EXPOSE 80
CMD [ "/usr/sbin/nginx" ]
I love a good rant as much as the next guy, but unfortunately, rants are rarely actionable.
> Maven, ivy and sbt are the go-to tools for having your system download unsigned binary data from the internet and run it on your computer.
The root of the problem is that out of the total number of libraries available in language X, only a small subset is packaged in Debian/RHEL. This may be more egregious with large, Java enterprisy software, but you could easily end up with the same problem in Ruby or Python.
You cannot reasonably expect developers to package and maintain all their dependencies properly. The least worse solution would be to:
- still use maven to manage dependencies
- create a Debian/RHEL package incorporating the dependencies (effectively vendoring them in the package)
Unfortunately, it is not that simple, because you need to make sure that your vendored-in-the-package dependencies are somewhere where they will not conflict with another package with the same idea and the same dependencies (or better, the same idea and a different version of the same dependencies). Which means you need to keep them out of /usr/share/java and make sure the classpath points at the right location.
However, it seems that developer tend to avoid this kind of rigmarole and instead go for the "install dependencies as a local user" for certain classes of application (eg, webapps) because packaging is not fun.
Maven packages aren't actually unsigned either. They're downloaded over SSL and to get into Maven Central you need to sign with a GPG key.
The problem is that you normally cannot find a path to the developers through the web of trust, of course, but that's not Maven's fault. That's the fault of the web of trust (more accurately called "handful of strings of trust").
Debian/Red Hat code signing doesn't prove very much either. All it proves is that the package came from Debian or Red Hat. But did they modify the software along the way, doing some kind of MITM attack on the upstream developers? Quite possibly! At least with Maven you don't have that problem.
The problem is that you normally cannot find a path to the developers through the web of trust, of course, but that's not Maven's fault.
It is Maven's vault. Debian could have taken the same approach, but instead they have decided to vet prospective developers for a few years, and use one signing key for package indexes.
As a result, in Debian:
- It is feasible to actually check packages, because you don't need a large WoT.
- You know that the maintainer of a package was a least vetted for some time before they could put anything in the archive.
With Maven:
- It's infeasible to check the signatures, since nearly no one has a complete enough WoT.
- Anyone can submit a library, no vetting necessary.
- Maven does not check signatures by default.
So, basically the only thing you can do in Maven is hand-pick a small set of libraries. Verify them in some way and stick to specific versions, putting their checksums in your POM. Unfortunately, as the article already touches upon, too many Java libraries pull in half of the internet, so it may not be very practical.
> It is Maven's vault. Debian could have taken the same approach, but instead they have decided to vet prospective developers for a few years, and use one signing key for package indexes.
Yes. But this creates a much higher barrier to entry. As usual, it is a trade-off, but it is obvious that with a Debian-like system, the number of packages on Maven/CPAN/Pypi/Rubygems would be considerably smaller.
I don't know any tooling that turns (recursively) a Maven pom file into a Debian repository of Debian policy-abiding .debs, which are magically updated when the pom file changes.
If you mean Debian policy-abiding in the sense of "signed by a Debian developer in the Debian WoT" then no, but that's not something you could ever do automatically. But for the rest, I've done all the individual pieces before: it is trivial to generate a .deb from a maven pom and put it in a debian repository, it's trivial to do some operation on all the dependencies of a maven project, and it's trivial to hook something into a maven repository to happen whenever a new artifact is uploaded. You absolutely could do this if you wanted to, and it wouldn't be more than a few days' work.
(Of course it wouldn't provide any value, which is why no-one does it).
If they are not public, that's obviously not the same thing, though you're still going through a lot more complexity than "here is this .war, put it on the server and reload the webapp".
> You cannot reasonably expect developers to package and maintain all their dependencies properly.
Why not? This is exactly what developers are expected to do. Every developer must manage dependencies for their application to work.
What you mean is they can't be reasonably expected to do it well.
In most situations, this is truly a trivial amount of investment relative to the overall cost of developing and maintaining an application.
There are ecosystems which make the task easier or more difficult/annoying, but that cost should be accounted for when choosing which development platform to use.
> In most situations, this is truly a trivial amount of investment relative to the overall cost of developing and maintaining an application.
I wouldn't say that creating and owning packaging for, say 50 libraries (assuming you only deploy to a single platform) represents a "trivial amount of investment" for a standard small developer team.
Honestly, if a particular development environment for a project required tracking 50 libraries not supported by any distro community and my development team was so small that I couldn't do that in a sane manner...
I might reevaluate the suitability of that particular language for the project.
On a side, it only takes marginally more time to build a package for a distro as it does to build the package by itself. If you've already got the build done, 3 experienced guys could knock out 50 packages in a week. Inexperienced (in this task), but competent, devs should be able to do it in 2 weeks.
The contract between operations and dev (as concepts, not as people) is in need of renewal.
To my mind, that was what "devops" was supposed to be, but it's been a bit of a dogpile in the years since the term gained popularity.
Systems are opaque to most developers, and many developers wish to make their software opaque to the system on which it runs. This is a failure on behalf of our entire profession, not any one group.
Infrastructure software is in a bit of a renaissance period, but it's very early days. Packaging software is a total mystery to most developers. I don't even need to back that up with examples, most of us can recall the last time we can across a well packaged piece of software with joy due to sheer rarity. I'd be very surprised to find the average age of a Debian maintainer was trending anything but upwards, and steeply.
Containers are being misused, but that's because the alternatives we've been building for ourselves have not kept up with the strong user experience narrative of web and mobile software.
It would be nice to have a well known 'devops manifesto'. I google'd it and came across this: https://sites.google.com/a/jezhumble.net/devops-manifesto/. Which I think is actually pretty decent - the emphasis on cross functional product teams, for instance.
In my mind, that is largely what devops is about - team ownership of the entire product, which includes infrastructure. Instead of having a silo'd 'ops' team writing ansible scripts and doing deployment, this should be part of the team (which could mean having an opsy guy on the team).
Anyways, as it pertains to containers, I think containers are more a practice than a principle. It tends to happen naturally when you want reproducible builds and continuous delivery. It's not really about making systems opaque to software, imo, but rather making your product artifacts reproducible (if you rely on running ./configure; make at deploy time, you never know what you'll end up with since dependencies are dynamically determined).
What is really ironic is that none of this "tools" solves any fundamental problem of so-called version hell and none of this containers are fundamentally different from
./configure --prefix=/xxxx && make && make -s install
with or without following
chroot /yyyy
The big "innovation" of having so-called "virtual env" (they call it "reproducible [development] environment) for each "hello world" (a whole python/ruby/java/etc installation with all packages and its dependencies in your [home] project directory) solves no real problem, only pushes it to the next guy (what they call devops).
Some idiots even advocating to have a whole snapshot of an OS attached to your "hello world", and even to make it what they call "purely functional" or even "monadic" (why not, of someone pays for that).
Unfortunately, there is no way to ignore complexity of versions and package dependencies or easily pushing it to "devops". Creating a zillion of "container images" with just your "reproducible development environment" or a whole "OS snapshot" just multiplies entities without a necessity.
Programmer must be aware of which version of what API implemented with what version of package or library he using and explicitly assert and maintain these requirements, like all the very few sane software projects (git, nginx, redis, postgress) do.
btw, the GNU autotools (which gives us ./configure) is somewhat evolved real-world solution - you have to explicitly check each version of each API both at the compile (build) time and refuse to build in case of unsatisfied dependencies and at the install time (and package manager must refuse to install if case of mismatch). This is the only way back to sanity, however "painful" it is.
> Programmer must be aware of which version of what API implemented with what version of package or library he using and explicitly assert and maintain these requirements, like all the very few sane software projects (git, nginx, redis, postgress) do.
Except that when you're doing anything that looks like an actual end-user application (as opposed to infrastructure), you end up using dozens of libraries which themselves have dependencies, so suddenly you're supposed to "explicitly assert and maintain" hundreds of different library versions, none of which is in any way relevant to the application you're building.
I myself see Docker containers as the only reasonable way for giving a service application to people to deploy on their machines, because even the programming runtime I need is 5 years out of date on Debian/Ubuntu, and installing that stuff manually is a) pain, and b) different on every operating system.
That would be the same git that's still basically unusable on windows? And have you ever tried to cross-compile it?
> There is absolutely no fundamental problem with ./configure; make; make install.
The fundamental problem is incompatible versions of dependencies. Arguably it's in linux's dynamic linker rather than a problem with configure/make. But if you need to run something that depends on libfoo 2.3 and something else that depends on libfoo 2.4 on the same machine, you need something like docker.
I would be interested to know why you call purely functional package management "idiotic", given that one of its main goals is to solve the problem of version hell? I.e., to make it easy for a developer to specify that application X should use version V of library Y, without interfering with other applications on your system?
Very simple. There is so-called atomic operations, or transactions, which are good enough to solve the problem. Any "pure functionality" in this context is plain nonsense.
I think you are conflating purely functional, reproducible environments with the (for lack of a better term) Docker way of managing containers where you just make full disk image for everything with lots of duplication. It is very possible to have the former while avoiding the latter. Both the Nix and GNU Guix projects succeed at this. I recommend taking a look at them to see if they address your concerns.
I am old-school sysadmin and I still thinking in terms of what ldd command tells me, how shared libraries are implemented, how various dlopen based FFIs work, why I need this or that.
The other approaches, such as Java's (where we "abstract out an OS") in practice leading only to a bigger mess, because it boils down to the very same libc, libm, libffi and friends. JVM is an ordinary userlevel program, so it obeys to the restrictions and rules for any other userlevel program. This is the sad truth for Java zealots.
Basically, one cannot ignore an OS (at least when you still want to dlopen cand call the stuff instead of re-implementing it poorly) - it is just a wrong idea, leading to all these ridiculous FS-inside-JVM implementations and other messed up layers of unnecessary, redundant abstractions.
System administration is still hard and it (necessity to think, understand and analyze) cannot be eliminated by some bunch of shell or ruby scripts and wishful thinking.
465 comments
[ 3.2 ms ] story [ 303 ms ] threadhttp://en.wikipedia.org/wiki/Abstraction_inversion
http://en.wikipedia.org/wiki/Leaky_abstraction
However, installing from source with make was (and remains) a mess. It may work if you're dedicated to maintaining one application and (part of) its stack. But even then it usually leads to out of date software and tracking versions by hand.
Many people have this weird aversion to doing basic sysadmin stuff with Linux. What makes it weird is that it's really simple. Often easier than figuring out another deploy system.
(The neckbeard in me blames the popularity of OSX on dev machines.)
[1] https://nodesource.com/blog/nodejs-v012-iojs-and-the-nodesou...
While I agree with the articles main points - the GNU build system is far from simple. Basically an arcane syntax limited to unix-based systems and 5 or 6 100+ page manuals to cover.
It doesn't excuse it - but I think it's easy to see why people turn to curl | sudo bash as the author puts it.
It doesn't do dependency management though, which is an externalised cost. But that's what rpm/deb do.
I see the attraction of containers and disk image based management. It's much less time consuming. But it's very much the opposite of ISO9001-style input traceability.
"Usually" indeed. Because if it breaks, you do need to know the implementation details to figure out what's wrong.
The problem is installing 206 different pythons on my system just makes it more likely that something else is going to break.
ISO9001 often turns into its Dilbert parody of bureaucracy, but the core ideas are sound: if you have some sort of failure of production, it's useful to know what went into the production process and where it came from. So in the case of deploying images, then yes: you get repeatable copies of the image. Provided you know where the image came from. Images themselves aren't usually stored in a version control or configuration management system. It may not be obvious where the image came from. And, if an image is made up of numerous "parts" (ie all the installed software), you need to know what those parts are. If an SSL vulnerability is announced, what is the process for guaranteeing that you've updated all the master copies and re-imaged as necessary?
A few years back we had an issue where a mysql script was over the limit for phpmyadmin - my fairly experienced colleague he was unaware that you could log into the cli and use mysql from the cli.
The same reason people prefer to use garbage collected and dynamically typed programming languages.
It is merciful that GUI environments are immune to these deficiencies.
Which is why there's alternatives – cmake, waf, …
Gah. Because I want to drop a metric ton of python code into my own source tree just to build. (gnulib is bad enough...)
Personally I like make. I understand it. I've used it for something like 20 years now. If there are problem domains it doesn't work for, they aren't problem domains I encounter. (Like so much of Linux software in the past 5 years or so, I find myself saying "this seems like an interesting way to solve a problem I simply don't have".)
Each their own poison. Personally I don't like them either, but pretending it's autoconf or curl|sh is an oversimplification.
I managed to get haddoop running on a small cluster from scratch Michael Nolls turtorial is a good starting point.
Full stack should mean you can and have used a soldering iron in anger and also have at least a CCNA level of networking.
Edit: Maybe I should have added a /sarcasm to my comment?
I think a lot of this is a result of what I like to call the "Kumbaya approach to project/team management":
This is where you have a team (either for a single project or a team at a consulting agency, etc) that is effectively all development-focused staff, possibly with some who dabble in Infrastructure/Ops. In this environment, when a decision about something like "how do we get a reliable build of X for our production server deployment system" needs to be made or a system needs to be supported, no idea is "bad", because no one has the experience or confidence to be able to say "that's a stupid idea, we are not making `curl http://bit.ly/foo | sudo bash` the first line of a deployment script"[1]
[1] yes this is an exaggeration, but there are some simply shocking things happening in real environments, that are not far off that mark.
Edit: to make it absolutely clear about what I was referring to with [1]:
The specific point I was making was running something they don't even see (how many people would actually look at the script before piping it to bash/sh ?) from a non-encrypted source, and relying on a redirection service that could remove/change your short url at any time.
Unfortunately I was stupid enough to ddg it (duckduckgo it, as opposed to google it) and apparently this exact use-case was previously the recommended way of installing RVM[2]
[2] http://stackoverflow.com/questions/5421800/rvm-system-wide-i...
-
I'm totally stealing this :)
No, it's not :(
It's really really bad, but people will continue doing it until commands/things become so easy we can actually understand what we're doing. Unfortunately, this has never been a priority in Unix-land as far as I've gathered.
tar c . | gzip > /tmp/out.tar.gz
Part of the problem is that each command line utility has its own flag language, and equivalent functions often have different letters. For instance, very often one command has "recursive" as "-r" while another has it as "-R". It's impossible to remember it all unless you're a sysadmin.
(I'm a developer.)
Sequences of commands sometimes get pasted into a conveniently-located text file; if I find myself repeating the operation I might turn it into a script, a shell function for my .zshrc, or an alias.
Just 10 minutes ago: mysqldump [args] | nc -v w.x.y.z 1234 nc -v -l 1234 | pv | mysql [args] (after an initial test that showed adding "gzip -1" was slower than uncompressed gigabit ethernet.)
Except with cp , -R is the safe one and -r is the dangerous one. And there are tons of little inconsistencies like this.
As a former sys admin, I did that all the time. Who the hell can remember how to convert an SSL certificate to load it into a Glassfish app server? Didn't mean I couldn't step through all commands and figure out why it did that before I loaded the new cert... And next time, I just need to go to my quick hack repo for the magic incantation.
On a Unix based system, tar is just used so frequently and for so many purposes, that not understanding it feels a bit like working in a shop and not knowing how to use a roll of tape.
"regexsearch" is more work to type and more space taken up everytime 'grep' appears in a command-line. And says nothing about recursion.
Edit: the rest of my comment (somehow submitted to soon!)
(I can imagine some sort of hackery that determines if less or something is running and scrolls that, but it sounds like a huge mess. Is that actually what you're doing? Does it send keypresses? What if you're in a mode where those keypresses do something besides scrolling?)
I'll be honest - I have ~no idea~ (edit: apparently there are xterm control sequences for mouse scrolling) how it's actually implemented, but several tools have some reference to mouse support (tmux, vim, etc) in option/config files, so it's probably available for your distro/platform and just needs to be enabled.
Further edit: (or PS. or whatever):
`less` pager supports mouse scrolling. `more` pager does not!
1. I routinely need to look things up that are a bit murky in the deep recesses of my memory.
2. I am reminded continually of how nice it is to have man pages that are well written, are easily searchable, reference appropriate other pages, and are helpful enough to remind you of big picture considerations that you didn't realize you were facing when looking for a commandline flag.
Google query: git display file at revision. Immediate answer (without even having to click any links, it's in the result description): `git show revision:file`
Total time: 5 seconds
Trying to reproduce with man and help:
search for display, finds nothingstart scrolling down
notice git-show (show various types of objects); sounds like a likely candidate
..no output .. useful OPTIONS <object>... The names of objects to show. For a more complete list of ways to spell object names, see "SPECIFYING REVISIONS" section in git-rev-parse(1). a lot about specifying revisions, nothing about how actually specify a fileGive up. Google it.
When I google something, I usually do not remember the answer to my question, the only thing I remember is the keyword to put in my futur query to get the same answer. You will get your answer quicker, but you wont learn much. So personally, I prefer reading man pages (when I can) than use google.
Rsync for example, where trailing slashes make a difference and it's not obvious from skipping over the manual.
Looking at working code/commands often works better than piecing it together from the manual imo.
But it isn't all that hard to understand a clean Unix. I have never copied or typed a command that I don't understand.
One problem may be that most Unices these days is not as clean anymore as, say OpenBSD or NetBSD. E.g. the recent X stack, with D-BUS, various *Kits, etc. is quite opaque. This madness was primarily contained to the desktop and proprietary Unices, but seems to spread through server Linuxes these days as well (and no, this is not an anti-systemd rant).
Well, good for you. I can assure you that it's not the case for almost anyone who approached Linux after the likes of Mandrake were released and/or tried to make it work on anything different from a traditional server.
I'm all for trying to understand what one is doing (and I wholeheartedly agree with TFA's point), but the reality is that very few people in the world really understand all intricacies of one's operating system. This does not excuse poor security practices, but it explains their background.
You wouldn't hire some high school kid who's just about taught themselves HTML by reading a book for a week, and get them to write your web application from ground up. You'd hire someone who knows what they're doing. Why is it seen as any different for Operations work? There is a reason systems administration is a skilled field, and a reason they're paid on a par with developers.
However yes the issue of a team that "doesn't make money" is very real. Maybe you it should be "marketed" like legal or accounting: it doesn't make money, it saves money caused by SNAFUBAR situations.
1. I saw it done that way in some blog.
2. We did it like that at my last job.
3. Seems like it works.
/(apt-get|yum|dnf) install (apache2|httpd|nginx|lighttpd)/
One of the problems (as I tried to argue) is that most Unices have become far more complex. The question is if the extra complexity is warranted on a server system, especially if bare Unix (OpenBSD serves as a good example here) was not that hard to understand.
Of course, that doesn't necessarily mean that we should look back. Another possibility would be to deploy services as unikernels (see Mirage OS) that use a small, thin, well-understood library layer on top of e.g. Xen, so that there isn't really an exploitable operating system underneath.
To note that it's trivial to change what goes into the clipboard too. Copying and pasting commands from potentially untrustworthy sites should be ruled out too, even if understood
This because they want to retain their ability to shop for off the shelf hardware, while getting away from a platform that has proves less than functional for mission critical operations (never mind being locked to a single vendor).
What seems to be happening is that there is a growing disdain for power users and "admins". The only two classes that seems to count are developers and users, and the latter needs to be protected from themselves for their own good (and developer sanity).
When you read the script in a browser, than pastes it in a terminal, you know that "scp -r ~/.ssh u@somehost.com" isn' there.
http://thejh.net/misc/website-terminal-copy-paste
For example, it won't protect against stealing the .ssh folder and installing a keylogger at your computer.
Not only "previously", it's the current recommended way to install rvm. From their front page:
>> curl -sSL https://get.rvm.io | bash -s stable
[1] http://rvm.io/
https://rvm.io/rvm/security
The whole security, whether GPG is invoked or not, relies on the security of the HTTPS connection alone.
If the HTTPS cannot be trusted alone, then everything is lost as a compromised HTTPS connection can be used supply both a comprimised GPG key and a compromised package, or, indeed, anything at all that is legal to `| sudo bash`...
And HTTPS security boils down to:
1. The difficulty of altering (or exploiting privileged position wrt) the global routing table to setup MitM or MitS scenarios.
2. The difficulty of obtaining a valid looking certificate for an arbitrary domain.
Any situation where a government actor is the adversary poses intractable challenges to both 1 and 2 above. (And before you say NSA/GCHQ would never care about XYZ, consider China...
So even if you trust https works, it's still a tricky proposition - it's not really similar to a distro's package distribution channel.
[1] http://get.docker.io/
[1]: https://github.com/discourse/discourse/blob/master/docs/INST...
Are you really trying to say that the instructions for installing Docker should be considered in-scope for a guide to install Discourse on a cloud server?
They've included a short snippet that will get you a Docker, in the way recommended by Docker, for whatever your base system is. Many production systems do not move at the pace of Docker development, so it's not practical to run Docker from your distribution's package archive. Some distros will not have distributed packaged Docker releases at all.
What's wrong with these instructions? If you are really "dealing with this" right now, it is worth noting that something like 20 or more supported platforms have specific Docker installation instructions from the Docker website.
https://docs.docker.com/installation/#installation
From a quick sample of those instructions, only the Ubuntu instruction page uses the wget|sh method, and it's using an SSL connection to Docker's own website to add an apt source with signatures in the supported way. This way should work on any Debian-based or Yum-based distro, and writing the instructions like this must likely save Discourse from getting a lot of "How do I docker" issues and e-mails from their clueless users.
So, would you prefer that part just says "installing Docker is out of scope" or should the Discourse developers go through every distro and cloud system and document the specific instructions for that? To do that would completely defeat the purpose of even using Docker at all.
Quick directions like these aren't questioned by users who just want to get things done, and they invite security risks just as the parent & article suggest.
I understand that curl pipe sh could have security problems but I also don't see it as that much different than the "normal" and "ok" way of doing things. I would consider something like the below pretty normal.
Because of familiarity, we aren't going to be too worried about what we are doing. If we are on a secure system (like a bank or something) then we've probably already gone through a bunch of hoops (source check, research) and we mitigate it like anything else.What is so different about
We didn't check the md5s in the first example, so yolo, we don't care about the content of the tarball we just `make install`-ed. We're assuming the webserver isn't compromised and that https is protecting the transfer. Is it because the tarball hit the disk first? Does that give us a warm fuzzy? Is it because "anything could be in installer.sh!!!?! aaaaah!". Well, anything could be in Makefile too right? Anything could be in main.c or whatever.I agree that curl sh | sudo bash makes my spidey sense tingle. But if I really cared, I would read the source and do all the normal stuff anyway. So I think it's some kind of weird familiarity phase we're all in.
A distro (or official vendor, or possibly a trusted third-party) repo of pre-built, signed packages would always be my first choice.
If one of those isn't available, my next step would be to create a package for the tool in question, part of which is setting up a file for `uscan` to download new source archives, and compare against the signatures.
In this scenario we (as in the organisation) are now responsible for actually building and maintaining the package, but we can still be assured that it's built from the original sources, we can still install it on production (and even dev, staging, whatever) servers with a simple call to apt/aptitude, and dependencies, removal, upgrades, etc are still handled cleanly.
I find it interesting that binary packages have existed for decades and yet `rpm etc` knowledge is rare. Why did curl sh become popular? Why doesn't every project have rpm|deb download links for every distro version? Why don't github projects have binary auto-builds hosted by github? I'd argue that it's too difficult. Binary packaging didn't succeed universally. For deployment, containers are (in the end) easier.
But the original article is conflating container concepts and user behavior (not wrongly). If docker hub does end up hosting malware-laden images, it would be interesting emergent behavior but it would be orthogonal to containers. Like toolbars. Toolbars probably aren't evil. A vector for evil maybe?
What makes you think the knowledge is rare? Among developers who actively target linux distributions I would imagine the opposite is true.
Even a number of the referenced curl|bash offenders are just using that as a "shortcut" to add their own apt/yum repos and calling apt-get/yum to install their binary package(s).
Or, you know, maybe someone updated the whatever.io installer to make it 'better'. But you are trying to debug some problem and you made one image last month and another one this month and you're pulling your hair out trying to figure out why they are different. Oh, it's because some text changed on some web site somewhere.
You've taken a mandatory step and put it outside your sphere of control.
With a good build system (eg. autotools) writing an RPM spec takes almost no time at all and if you have the proper infrastructure in place for building packages, you can have something workable in a very short time.
Self-packaged RPMs also don't need to be quite as high-quality as ones you might want to include in a distribution, so if it makes sense for your use case, it's perfectly okay have "bloat" (eg. an entire python virtualenv) in your package.
Yikes. Have we sunk this far?
* Using checkinstall to create a local deb/rpm which can be easily installed/removed later instead of "make install". * What if installer.sh says "rm -rf /tmp/PACKAGE-build" and the connection is interrupted just after the first "/", you now have "rm -rf /". Oops. * configure will tell you what files it needs, and apt-file will tell you want dependencies to install. * I know what make install does. I know make. Who wrote installer.sh? Do they know anything about writing good software? Steam wiped out home directories, who knows what these people do
Like, most of the way deployment should work is that you come up with some collection of packages that need to be installed and you iterate through and install them. Bob's your uncle.
Thing is, all the packages you need live out in the wild internet. Ideally, you'd just be able to take a package, vet it, and put it in your local artifact store and then when your production deployment system (using apt or yum or pip or gems or maven or whatever) needs a package, it looks at your local artifact store and grabs it and goes about its business. Never knowing or touching the outside world.
And your developers would all write their apps to deploy through the normal packaging methods that everyone and their mother is already familiar with and they could just put them into the existing package index as well.
But you've gotta lay out pretty serious moola (from when I last looked into available solutions to this) or set up a half dozen different artifact stores if you want to do things that way. And good luck managing your cached and private artifacts if you do. And on top of that developers don't necessarily know how to set up a PyPi or a RPM index or whatever so that the storage is reliable and you've got the right security settings or whatever else. (I know I sure don't and I'm not really interested in reading all of the ones I'd end up needing).
Setting up RPM is shockingly easy. It can get more complex, but the basic system is:
That will create a repo from all the .rpm files in the REPOBASE. Also you will of course need a GPG key pair, but that can be generated with `gpg --gen-key` where you give it a description of "Sign Repo" (or change the above commands to the key description you used).Then you get to decide on the deployment machine if you want to trust the repo (or you don't trust any and import the key via some other process, aka direct gpg import).
Of course you can find a bunch of more detailed explanations with $SEARCHENGINE, but if it takes more than a day to figure it out, your doing something wrong.
Building a set of RPM's isn't that much harder if you have a proper build system. But these are the kinds of things you give up when you decide to grab the latest immature hotness created by someone on their day off.
I don't think the problem is as rampant as it's made out to be in TFA... that said, most people don't look at said script(s), so it's entirely possible something could have been slipped in. For that matter, I think the issues outlined in the article relate more to overly complicated Java solutions (the same happens in the .Net space) that are the result of throwing dozens of developers some with more or less experience than others at a project, and letting a lot of code that isn't very well integrated slide through whatever review process does or doesn't exist.
I see the issue with doing it for the general public ala RVM, but internally where you control everything I don't see the issue with curl into sh.
This is an obvious example:
http://en.wikipedia.org/wiki/Inner-platform_effect
Obviously a config / deployment system, like any other system, will start small and simple and "save a lot of time" but after an infinity of features are bolted on, it'll be infinitely worse than just using a bash script. Even worse, you probably figure out your deployment by hand on one system using bash, then need to translate what worked on a command line into crypto-wanna-be-bash config system (probably creating numerous bugs in the translation) then using wanna-be-bash to slowly poorly imitate what you'd get if you just used bash directly...
The last straw for me was trying to integrate some freebsd servers and /usr/ports had like six versions of cfengine none of which worked perfectly with the three versions on the legacy linux boxes. Screw all that, instead of translating bash command line operations into psuedo-bash I'll just use bash directly. IT is an eternally rotating wheel and the baroque inner platform deployment framework has had its day... and being an eternally rotating wheel it'll have its day again in a couple years. Just not now.
Not throwing the baby out with the bathwater, a strict directory structure, and modularity and library approach to error handling and reporting and logging which you can steal from the deploy systems is a perfectly good idea.
Unix philosophy of small perfect tools means I'm using git instead of my own versioning/branching system, and using ssh to shove files around rather than implementing and static linking in my own crypto and SSL system.
I'd prefer something like scsh (or a Common Lisp or elisp version thereof) for this sort of work: access to a full-fledged programming language and easy access to the Unix environment.
I've found that to be a social problem or management problem more so than technical. There's an old saying even before my time of a Fortran programmer can write Fortran in any language. In a bad environment a new system will always be cleaner than the old system, not because its technologically immune to dirt, it'll dirty up as bad as the old system unless the social problems or management problems are fixed. You really can write read only Puppet scripts. Or you can write readable bash. Or even Perl.
Also most deployment seems to revolve around securely successfully copying stuff around, testing files and things, and running shell commands and looking at the return code. Shells are pretty good at running shell commands like those in a maintainable easily readable and troubleshootable fashion. Its possible that a deployment situation that more closely resembles a clojure koan than the previous, might have some severely blurred lines. And there's always the issue of minimizing the impedance bump between the automated deployer and the dude writing it (probably running commands in a shell window) and the dude troubleshooting it at 2am (by looking at the deployment system in one window and running commands in a shell window next to it to isolate the problem). I would agree that cleaner library/subroutine type stuff in shell would be nice.
And you are correct, scsh is really cool but two jobs later some random dude on pager duty at 2am is more likely to know bash or tcsh. Principle of least surprise. I suppose if only scsh guys are ever hired... Then again as per above most deployment is just lots of moving stuff around and running things so its pretty self explanatory. But if the work is trivial, don't deploy a howitzer to swat a fly.
Maybe another way to look at it is if you're doing something confusing or broken, plain common language will clear things up faster and more accurately than using an ever more esoteric domain specific language. Or some folk saying like "always use the overall simplest possible solution to a complex problem".
There is the "don't reinvent the wheel" argument. I have a really good network wide logging system, a really good ssh key system for secure transfer of files, a strong distributed version control system to store branches and versions, a strong SSL infrastructure, a stable execution environment where upgrading bash probably won't kill all my scripts, a strong scheduled execution system... I don't need a tight monolithic collection of "not so good" reimplementation of the above, running that is more painful that rolling my own glue between the strong systems I already have. And using the monolith doesn't mean I get to abandon or ignore the "real" strong infrastructure, so the only thing worse than running one logging/reporting system is having to admin two, a real enterprise grade one and a deployment-only wanna be system. I did the puppet thing for many years. So sick and tired of that.
I'm pulling my hair saying are you even aware of what you're doing?
Go look up how you install snort or bro on centos. You have to either install from source, or install from a rpm from there website which may or may not have issues. This means you lose dependency management, and update management. Pure madness
I'm not able to go full docker on my machines @work, but I do have some statically linked tarballs. There is a reason apps that deploy in hostile environments (skype, chrome, firefox) bundle most of their dependencies.
1. Run this totally opaque command which might DTRT, and might completely pwn your system.
2. Prepare for 4 hours of dependency hell.
The advantages are that it's easy and you can make it work on almost all unix-like systems out there.
The only disadvantage is that you have one additional weak point: The server can get contaminated. Before you had to contaminate one of the many developer machines / build machines.
The situation hasn't been better before. Install media always got downloaded without ssl encryption or any certificate checks. This is still the same, but at least you won't get a hacked kernel today if you use secure boot.
To pick one example why...
Just because it's easy to run doesn't mean it's easy to support or maintain. Chances are `curl | bash` scripts aren't designed for your particular OS, so it's yet another form of software that you have to learn how to update, as opposed to using the OS-level update mechanism, such as yum, apt, or even brew to some extent. Being a good sysadmin doesn't stop at installing the software. Most of the hard (boring) work is in maintaining systems and keeping them updated and secure. Blind install scripts make this job impossible.
There is a very big difference between installing something on your dev machine to just get it started and deploying something into production. `curl | bash` is okay for setting something up on a dev machine where the only one that needs to use it is you. For productions machines, it's completely inappropriate[1].
[1] This is somewhat mitigated by things like Docker, but I'd still argue that you don't want to have an ephemeral installation method for containers either. You should have fixed versions that are installed by either a package manager or at least a Makefile.
Even assuming the URL's publisher is trustworthy (which is a poor assumption to make, ever), you forget SSL / HTTPS is broken, that the NSA has established MITM on the entire internet, that your installation process (which should be both versioned and repeatable) now has zero versioning and all the entropy of the network plus bonus entropy.
https://twitter.com/kylegordon/status/590860756075294721
And at that point kylegordon had earned it.
Like developers who won't write SQL and insist on an ORM.
The question, therefore, is: what is the simplest alternative installation method for OS X and Linux?
A container is a container, as long as docker itself has not bug, the container can only harm the containers content.
Most problems exists in the custom created software in the container (e.g. web-services with bugs, backdoors, ....), this will be a problem for Docker, VMs, Real-Servers, whatever too.
The real problem is the interoperability of different container, if you link the whole data, without any audit, to another container, you can have a problem, but this problem is not docker specific.
Presumably a container has network access of some sort? Malicious code could start probing and attacking anything exposed that way.
>> this will be a problem for Docker, VMs, Real-Servers, whatever too.
The implication is that you wouldn't get into this situation with a 'Real-Server' so easily, because you wouldn't just download an image and run it, without having an update/patch strategy or having much more idea of what's going on inside it.
No, I'm presuming it has some sort of network access, a malicious container could (for instance) still probe other containers for vulnerabilities, serve malware etc etc without full network access.
>> A firewall must be configured, but a firewall must be configured for a VM too. My point is, that their is not so a huge difference for production systems.
If you're downloading VM images from somewhere and running them without checking what's in them you'll run into the same problem, sure.
The problem being pointed out here is that when applications are bundled outside of the purview of a packager like debian you -
- Being a backdoor to the rest of your network (sniffing network traffic, or more simply reverse ssh-tunneling to an outside server)
- All the various "fun" botnet-related activities (spam being the king here)
- Actively serving malware to the rest of your network.
EDIT: Formatting, and well, others answered your question less specifically but more eloquently.
So given that there are no bugs and as long as the Linux kernel is free from local privilege escalation exploits. That seems long odds to trust in.
One of the criticisms in the article is that much of what's going on now, either with containerisation or weird build systems like Hadoop's, misses out on this.
YOU ONLY LIVE ONCE MAN! trust the (maven) system
These are the same teams that have overprivileged accounts for the database or sudo-enabled users running applications or chmod 777 all over the place.
Even things like Chef cookbooks have this going on. If you want to build from source because it's not in your repository, then you're necessarily going to need to drag in sbt or gradle. (see https://github.com/hw-cookbooks/kafka/blob/develop/recipes/d... as an example). Sure you could figure out the mirrors and download the correct binary from the website. You could also use this recipe to compile everything and then package it up to host yourself. (Both of these actions require writing custom recipes). Not everyone has time to do this, and this magical recipe you found online works great on the development server! Just add it to the production server and now we've just used sbt in production on a software team.
Maven, on the contrary, is the biggest single source of signed packages around. Every package in maven central has a GPG signature - the exact same gold standard that Debian follows. The problems Debian faces with packaging Hadoop are largely of their own making; Debian was happy to integrate Perl/CPAN into apt, but somehow refuses to do the same with any other language.
> Instead of writing clean, modular architecture, everything these days morphs into a huge mess of interlocked dependencies. Last I checked, the Hadoop classpath was already over 100 jars. I bet it is now 150
That's exactly what clean modular architecture means. Small jars that do one thing well. They're all signed.
Bigtop is indeed terrible for security, but its target audience is people who want a one-stop build solution - not the kind of people who want to build everything themselves and carefully audit it. If you are someone who cares about security, the hadoop jars are right there with pgp signatures in the maven central repository, and the source is there if you want to build it.
The value of debian is not that they package (or repackage) everything into deb files but that they resolve versioning and dependancy conflicts, slip security fixes into old versions of libraries (when newer version break API/ABI), and make it possible to integrate completely disparate software into a system. They also have a great track record at it.
Maven does not do any of these things; Maven does nothing to protect the system administrator from a stupid developer, it just makes it easier for their code to breed and fester.
You must understand that the sysadmin has an enormous responsibility that is difficult for programmers to fully appreciate: You don't feel responsible for your bugs, you don't feel responsible for mistakes made by the developer of a library you use, and you certainly don't feel responsible for the behaviour of some other program on the same machine as your software, after all: Your program is sufficiently modular and scalable and even if it isn't, programming is hard, and every software has bugs.
But the sysadmin does feel responsible. He is responsible for the decisions you make, so if you seem to be making decisions that help him (like making it easy for you to get your software into debian) then he finds it easier to trust you. If you make him play whackamole with dependencies, and require a server (or a container) all to yourself, and don't document how to deal with your logfiles (or even where they show up), how or when you will communicate with remote hosts, how much bandwidth you'll use, and so on: That's what Maven is. It's a surprise box that encourages shotgun debugging and using ausearch features to do upgrades. Maven is a programmer-decision that causes a lot of sysadmins grief a few months to a few years after deployment, so it shouldn't surprise you to find that the seasoned sysadmin is hostile to it.
Maven has exactly the same capabilities as deb does - you can depend on versions, depend on a range of possible versions, exclude things that conflict and so forth. And it puts even more emphasis on fully reproducible builds (with the aid of the JVM) - in that respect it's closer to nix than apt.
> But the sysadmin does feel responsible. He is responsible for the decisions you make, so if you seem to be making decisions that help him (like making it easy for you to get your software into debian) then he finds it easier to trust you. If you make him play whackamole with dependencies, and require a server (or a container) all to yourself, and don't document how to deal with your logfiles (or even where they show up)
Wow, self-important much? Too many sysadmins seem to forget that the system exists to run the programs, not the other way around.
> If you make him play whackamole with dependencies, and require a server (or a container) all to yourself, and don't document how to deal with your logfiles (or even where they show up), how or when you will communicate with remote hosts, how much bandwidth you'll use, and so on
On the contrary, maven makes the requirements much simpler. I have literally one dependency, a JVM, so it can run on any host you like (no need to worry about shared library conflicts with some other application). It needs to download one file (shaded jar) from our internal repo, and execute one command to run it. That's it.
> That's what Maven is. It's a surprise box that encourages shotgun debugging and using ausearch features to do upgrades.
No, it's just the opposite. All the dependencies and project structure are right there in declarative XML. It's what make should have been.
When make was written most machines would have just exploded at the sight of a typical build.xml, and downloading tens or hundreds of packages from anywhere was simply out of the question.
Also, 'dependency' means something completeley different in make as opposed to maven - I don't think modern build systems do even care much for make-style deps.
Sure. But the notion of doing things declaratively existed (Prolog predates make by five years). And the biggest difference between make and the scripts that preceded it is that it's more structured, with a graph of targets rather than just a list of commands.
If you add the ability to reuse libraries of targets (something that sort-of exists via implicit make rules), restrict targets to something a little more structured than random shell commands, and - yes - add the ability to fetch dependencies (including target definitions) from a repository, you end up with something very like maven.
As such I'm very pro containers as they will eliminate a ton of deployment effort and allow me to manage different environments much more easily. But it means that there needs to be a much bigger magnifying glass on the container contents early in the process as opposed to the moment of deployment.
You may feel that developers are some kind of loose cannons who don't care about quality and Debian is some kind of gold standard. From the other side of the fence, we do care about the quality of our software and Debian is a disaster zone in which people without sufficient competence routinely patch packages and break them. I specifically ask people not to package my software these days to avoid being sucked back into that world.
As a sysadmin you shouldn't even be running Maven. It's a build tool. The moment you're running it you're being a developer, not a sysadmin. If there are bugs or deficiencies in the software you're trying to run go talk to upstream and get them fixed, don't blame the build tool for not being Debian enough.
And then state that you don't want your own software packaged. So, if a sysadmin is not allowed to build and not allowed to package, how are they supposed to get your code into production? "curl foo | sh"?
One of the things I loved about my move to linux and .deb land was that if I uninstalled something, I knew it was uninstalled. I didn't have to rely on the packager remembering to remove all their bits, or even remembering to include an uninstall option at all. Or rely on them not to do drive-by installs (which big names like Adobe still do, out in the open). And not have every significant program install it's own "phone home" mechanism to check for updates. The crapstorm that is Windows packaging is a fantastic example of a place where developers love and care for their own product, but care not a jot for how the system as a whole should go together.
Debian feels like a distribution maintained by a bunch of sysadmins: People who have shit to do, and who understand that the purpose of a machine is to get stuff done, not to run some software.
A lot of sysadmins believe since they are responsible for the software, they need to be able to build stuff and fix some stuff themselves (i.e. it can't wait for upstream). In my experience, it's usually something stupid (like commenting out some logspam), but it's critical enough that I can imagine a lot of shops making it mandatory to ensure they can do this.
Really proactive sysadmins do try to run fuzzers and valgrind and do try to look for bugs rather than waiting for them to strike. And sometimes they get it completely wrong, as in the OpenSSL/Valgrind disaster, but they usually ask first[1].
Now I don't agree with everything Debian do, and I don't want to defend everything they do, either, but I think programmers in general need to get out a certain amount of humility when dealing with sysadmins: Because when these sysadmins say that they're not going to package hadoop because the hadoop build process is bullshit, it isn't appropriate to reply "well you guys fucked up openssl, so what do you know?"
One thing that would help is if we didn't look at it as Programmers on one side of the fence and Sysadmins on another side. Programmers have problems to solve, and sysadmins have problems to solve, and maybe you can help each other help solve each other's problems.
[1]: http://marc.info/?l=openssl-dev&m=114651085826293&w=2
The analog of the aspects of what Debian does that you're talking about here in the HStack world are companies like Cloudera, who, surprise, make their stuff available as debs and PPAs.
Building your own Hadoop from source and complaining that the resulting product is unvetted is sort of like doing a git pull of all the Linux dependencies and building that.
Why should we assume the Debian Foundation is the sole trusted source of every type of software?
And yes, if I'm using Debian and didn't add any PPA or extra sources, then the Debian Foundation IS the sole trusted source of software. And you do that because you know hey won't fuck up the system, which (and that's the whole point of this thread) the others certainly don't.
Now Debian is telling you: we see now way to distribute this software and guarantee what you are getting or that it won't fuck up the system.
Do you think I'd consider installing that junk?
I could sign anything I like, but that doesn't make it any more secure for you to curl it into /bin/bash.
Signatures are about who signs it, and that's not something mvn has solved at all. Mvn is a free-for-all of binary code that very well could own my system, wheras debian is a curated collection of software which the debian maintainers have signed as being compiled by their systems with no malign influence and having met at least some bar.
I'd trust foo.jar signed by debian over foo.jar signed by bobTheJavaBuilder@gmail.com anyday... and mvn only gives you the latter.
So yeah, sure, they're signed, but it doesn't actually matter if you don't take the time to hook into the chain of trust (and believe me, mvn does not ask you to trust your transitive jar dependency 50 down the line) or have a trusted third party (debian) do their own validations.
And not only that, by shipping the source and requiring that binaries can be built from the source, who signs it is no longer blind trust. Others can audit it.
Reproducible builds should improve this even further.
This comes with a huge tradeoff, and I guess it's that tradeoff that makes developers like myself opt to sometimes even pipe the cURL to bash. I almost never download any software I actually plan to use through official system repositories, because whatever comes out of apt-get, it's almost always two years behind the last release and missing half the features I need. Sure, I'll apt-get install that libfoo-dev dependency, because I don't care what version it is as long as it's from the last decade. But for any application I actually need to use, it's either git repo or official binary download.
As a sysadmin, I love that, but I've had to come to terms with the fact that some developers have the attention span of hummingbirds who had Cap'n Crunch for breakfast ("two years old" is still very new software from an administration perspective).
So, I've basically accepted the fact that whatever stack the developers use I'll build and maintain directly from upstream -- with the price being that the version and extensions/whatever used are frozen from the moment a given project starts.
Oh, the "unix" philosophy.
It's not even a security thing. I've had poorly-written Makefiles that would have blown things away thanks to an unset variable on a certain platform, for example.
Can I read a Makefile? Sure. But 90%+ of Makefiles these days are 12000 line automatically generated monstrosities. It's not worth my time to bother opening the Makefile in a text editor in case it isn't, and I'd be amazed if many people did.
make -n you can do I guess. But unless you're also auditing all the source code I'm not sure there's a lot of value in it.
> It's not even a security thing. I've had poorly-written Makefiles that would have blown things away thanks to an unset variable on a certain platform, for example.
Yep. Maven doesn't do that.
So, with regards to your assertion, the second clause (support contract)? Definitely. the first (code review), never.
That's the view from my side of the fence, anyways.
The only place I've ever seen (or even heard of this) being done is banks and defense.
It ain't in ISO27001, and so nobody cares.
I've worked in places where there's a restricted list of 'approved' open source stuff, but that's been more to do with licensing.
I'll queue this to read later, but just reading the executive statement, it seems to jive with a lot of the discussion here.
The issue isn't that code review /shouldn't/ be taking place, or even that there aren't directives stating that it should be done. It's that it isn't being done, and that's a problem.
Eh, no.
If Basel requires that level of auditing, I'd love to see a cite. There are some pretty heavy-handed regulations out there, but this would be clearly unworkable.
I've never seen what you describe. And one of the companies is a huge multinational.
I wonder what laws you're referring to that requires companies to have read the innards of products they deploy.
Even the "getting a support contract from someone who does". Do you think for example Red Hat has read every single line of code shipped in RHEL?
Also which law do you think they would be breaking by not reading the source code of products used?
Reading the source code might be the case if you are in a large enterprise that can afford to keep those programmers busy or needs to actually vet the code but I honestly doubt it. Just check the loc count on something as common as Hadoop (close to 1 million [1]) and that is excluding the "stack" described in the article. I wanna bet you, because we all know how volatile the job market is that even the guys building or supporting the software did not write or read even half of that code.
[1] https://www.openhub.net/p/Hadoop/analyses/latest/languages_s...
Except they don't do that at all?
If a bug in git would cost your company a lot of money, why do you think GitHub would be accountable if you have some enterprise deal there?
I still have a hard time envisioning anyone in any position of authority in any credible company accepting the idea of installing open source software without vetting it one way or the other.
Well, that is an entirely different thing. People TEST stuff before deploying it. But that has little to do with code reading or mandatory support contracts.
I've worked in multiple large companies and even getting package signing turned on requires a lead pipe. Docker and tools can enable an org to move those types of responsibilities "over/down" to the developer as well, so that now there is no neck-beard encrusted gate at all.
I've unfortunately had more than one mind numbing conversation where another administrator will tell me that it's a "script" and therefor, I don't know what it does, and that there "may be" some hidden black magic that will instantly pwn all of our systems. Attempting to explain that I verified the functionality line-by-line brings blank stares, as if it's utterly impossible for someone to derive the dark magics that are code.
Thankfully, I haven't run into many instances where software is blindly installed as the author relates, but his point in that "no one knows how it works" combined with the lack of attribution creates an environment of mistrust which greatly limits our ability to take advantage of open source software.
I've never, ever seen this happen.
There is no option for me, as a user, to build and sign my own image with my own pgp key afaik. My organization might already have a chain of trust, and docker is asking me to ignore that and just trust their signatures (which also only work on dockerhub as of docker 1.5... don't know about 1.6 because you can't use docker for at least a month after a release else security holes galore).
Docker did nothing to encourage signing containers. At 1.0 they had no capability to do any signature, verification, whatsoever. It's being added as an afterthought, and poorly.
If you look at the AppContainer specs, signatures (pgp based) were built in from the very beginning, it lets me create my own chain of trust (including incorporating other's keys), sign my own images, trust someone elses signature, does not trust the transport or storage medium, and has integration with the clients.
If you want to convince me docker cares, you're going to have to give me examples of where they didn't fuck up...
Tell me how I can use docker's tools to sign my own images, optionally trust my friend Alice, and securely download images that she uploaded to her own registry or dockerhub but signed with her gpg key without me having to trust docker inc.
To my knowledge, all docker has right now is doing a 'tarsum' of images which assumes the registry is trusted and, even given that, can be downgraded for backwards compatibility reasons fairly trivially.
I agree that security and provenance is a real issue in Docker. It is however being worked on, and it will be solved. Presumably we will end up with some sort of app-store like framework with proper signatures and verification.
Docker can't do everything at once. Give them a chance. The new version of the registry is a major step forward in this regard.
In the meantime, what you can do is take redhat's advice. Rather than using a registry to get your images, operate a download site which stores archives of docker images that you can import with `docker load`. You can then also store signatures and check them yourself.
Security is a real issue in docker and it is being worked on, but I don't think "give them a chance" is a justifiable response. They're not focusing on it strongly. They already should have focussed on it and didn't. Their entire codebase was written without a security design in place, so there's likely deep-seated refactoring that'll need to be done before any new security-related features should be trusted.
They're working harder on monetizing and pushing docker as a production-ready standard as far as I can tell... I can understand not doing security before functionality, but it absolutely should be there before 1.0 or before you encourage others to use your software.
Docker has already lost any chance of me trusting their security with their lack of focus on it and I don't think it's excusable.
And if I'm doing what you say at the end, why the hell would I be using docker anyways then? I can already turn a tarballed fs into a linux container without docker (ty lxc); I thought the whole point of docker was sharing images and building on them and ... and having massive security flaws. Right.
For a public facing system I would agree that means it's not ready for production use yet...
I find your last point a bit strange. We all know the Docker development experience is a lot better than raw lxc. I'm saying you can (and probably should) be more careful about provenance than the Docker Hub is. Note that there are alternatives to the Hub with better provenance stories e.g: https://access.redhat.com/search/#/container-images (from https://securityblog.redhat.com/2014/12/18/before-you-initia...) This might make things a bit more awkward than it was before, but it's still not the same as raw LXC.
I feel your anger and I think it's understandable, but that doesn't mean things won't get better.
I can't remember what the application was, but I've seen an application where the only installation instructions where for Docker. That's just plain silly.
My concern with containers is that the wrong people will use it. There is a ton of software out there with just barely runs and make all kinds of assumption about it's environment. I fear that rather than design better, more correct software, these people/companies will start packing up their development environments as containers (more or else) and just ship those. Of cause that's no reason to discourage the use of containers, we just need to be critical of what is inside them.
We've been doing some work with Elastic Beanstalk lately, and - while it certainly does one or two things that are extremely clever and useful - in the end it just feels like this bizarre mix of complete magic and incredibly convoluted arcana. Everything feels very out of our control and locks us into an ecosystem that considerably limits our choices and flexibility (unless we invest the time in becoming experts in EB, which really isn't particularly something we have the time for). And, as the author of this post says, the security ramifications, while orthogonal, are also deeply troubling.
> Maybe the author regularly does full audits of the source code of every package he downloads, and of course disassembles every executable and library in the underlying OS, but most of us don't.
What matters is that the source code is auditable. It only takes one person to investigate something suspicious, raise a flag and get it fixed.
This is certainly still true for Debian - not being able to build from source is considered a release blocking bug.
- trust your compiler and linker
- trust your tar extractor / package manager / whatever
- trust your editor
- trust your http library (or whatever you used to download/distribute the code)
This can be overcome with signing.
We're all well aware of how deep this rabbit-hole goes, however that doesn't mean that it's a good idea to throw all trust away.
But the deeper you go, the harder it is for malicious code to reside there. In theory it's possible, but in practice I'd like someone to show me some code somebody could have written into the toolchain a decade ago, without hindsight, which could still exist today.
Whichever way, it's clearly far tougher for a malicious actor to compromise a system by injecting something into a distribution ecosystem than it is to inject a signed-by-unknown-reputation binary-only package into the Maven ecosystem.
Building on that, could we find a fixed point where the OS builds the exact same bootstrap binaries that were used to bootstrap the OS to begin with? [1] That would give us even more confidence that the binaries we're using are as they should be. Interesting place for experimentation.
[0] https://reproducible.debian.net/reproducible.html [1] https://gnu.org/software/guix/manual/html_node/Bootstrapping...
This is not the point being made. Trust is often offloaded, say to the debian people, but it is present in most modern linux systems as a basic part of the setup.
>> There's no wisdom or security to be gained from the act of running "make", much less "make install".
'make' is brought up not because everyone should be running "make" or "make install", but because it's a standard and it is understood by many people. It's brought up in the context of hadoop because the hadoop build system appears to be just so complicated and non-standard, including pulling in untrusted sources from all over the place, that it is near impossible to set it up as a well-audited, standardised package.
Given this, it is likely to be a hive of vulnerabilities, either during the setup phase (if any of the third party servers gets compromised or MITM'd) or during deployment (that java VM it pulled in during setup is never going to get patched).
You were not doing sysadmin right. DESTDIR and checkinstall are two vital tools that are learned from the mistakes of make/make install.
Fully agree with this.
Maybe I am just only another grey beard grumpy developer, but the new generations that grew up with GNU/Linux instead of UNIX, bash the security of other OSes and then go running such commands all the time.
docker and any user in the group docker, or lets say, any user capable of sending commands to the docker daemon running as root - is root on that system.
docker -v /:/f -w /f yourimage /bin/bash -c "echo root:and:so:on > /f/etc/shadow"
It has to be said. Damn the containers and windowization of Linux.
Docker is merely a more extreme example of the "package everything with the application" idea...
Vendoring dependencies and static linking is quite popular in executables, not just docker. Dynamic linking and shared libraries seem to be becoming a relic, deservedly.
BTW, the extreme example of "package everything with the app" is the unikernel movement.
But docker containers at least usually provide a dockerfile that describes exactly how a binary image is built. You just clone the source repo, audit the few lines of build commands and then build your own private registry. It's nearly no more trust than following the instructions of README or INSTALL. Just because fools are pulling down pre-build images and running in their datacentre, doesn't mean that's what way you should do it. And the problem with 'old-school' sysadmins is they are often far too quick to reject new practices, citing tired excuses based on misunderstandings of the technologies.
Yeah I have. It's easy if you have already built your 'stack' to scale horizontally (which means you have at least 2 or more of everything in a HA or LB config). You rebuild against a fully patched base-OS container, spin-up, send some test load to it & validate, then bring into service. Repeat for rest of nodes at that tier.If you are trying to be an old-school sysadmin that expects to console or SSH in and run 'yum upgrade' or 'apt-get upgrade' your containers then you are doing containers wrong...
The old-school sysadmins I know scoff at Docker's idea of 'containers'. Linux containers were already a thing, and don't need an entire copy of an OS ported around with them. To them, containers are a way of enveloping a process to limit it, not a way of distributing packaged software. They may or may not be doing 'docker' right, but they certainly know what 'linux containers' are.
Docker, whether your view is good or bad, brings something more than just another container implementation to the table...
Neither do Docker containers. You can build off scratch and put the literal bare minimum you need in it. I've done it a few different times. It's rarely done because the time and effort almost never makes up for the complexity and cost, but if your old-school sysadmins are scoffing it's on them.
But I'll get off your lawn now.
A container is a chroot environment for running a service. Basically. It's perfectly possible to 'yum upgrade' or 'apt-get upgrade' them.
I think what you're trying to say is this is a bad idea because containers are not supposed to be managed individually. The magical fluffy dream of Docker is to never have to manage another individual OS again; just make one image, then kick it out to all your machines and make them use the new image. No resolving dependencies, no verifying individual file checksums, no upgrading or downgrading or conflicting different package versions for every server and service. Just do it once, and then push it out everywhere, and everything magically works.
Right?
Here's the thing: Containers don't do away with the idea that a developer might use a totally bleeding-edge piece of software to update one of your many apps, and that you may end up with multiple incompatible versions of software on your systems. In fact, Docker kind of trades on that as a feature. "Install any shit software you want and it'll never conflict with other containers!" But this lie is shown for what it is once you start looking at containers as individual physical machines.
Back in the day we had to 'yum install some-specific-architecture-and-version-of-this-package' on a particular machine to make that machine serve that software. You do the same with Docker, because you have a particular Docker container with a specific version of the package, and all the other packages and OS requirements in that container. You still have a one-off machine to install, maintain and troubleshoot. The only differences are it isn't physical anymore, and you perform updates on the image, not on the machine.
Just like you ended up with a machine (or three machines) with three different versions of BDB, you end up with three containers with three different versions of BDB. Instead of using 'apt' or 'yum' to install them, you write their Dockerfiles and build them, test them, then roll out their updated images. You do a lot more from scratch now because a Linux distro hasn't done the work for you, and so you also run into all the headaches that someone packaging software for a Linux distro usually runs into.
One of the worst things about the 'new devops way' seems to be the non-reproducibility of things like Dockerfiles. To build a Docker image we slap together whatever bleeding-edge files we had on date X-Y-Z, and whatever comes out at the end is considered 'production'. Ignoring the patches, the quality control, the stable released software distributed by distributions on reliable mirrors, and generally without tuning the software at all for the particular system you're running it on. Try to build that Dockerfile again in a year and suddenly it doesn't work the same as it used to, or a bug magically appears on your production system, or you have to apply a patch and now you need to unroll all the commands used for individual stages of the build process for a single package and figure out how to make it still work like the original was built, etc.
Containers by themselves are fine things to use. The problem is how they've effectively sold a lie to everyone that uses them: that there is no sysadmin work to be done. That don't worry, Mr. Javascript Dev, you too can build infrastructure and deploy it without learning the many lessons and best practices of an industry that has been here longer than you've been alive.
This wouldn't even be far from the truth if it was, say, a RedHat-built set of container images, or a Debian-built set of container images. Then at least there'd be an expert who's building software in a reliable uniform way and along a particular standard...
All your points about bad sysadmin practices are OS & container agnostic - they can happen on any platform, don't drag Docker into it. Sure there is a culture of 'docker run somebinaryimage [..]' but those people are the ones that do "curl | sudo bash" as well.
Your claim about non-reproducibility of Dockerfiles is bogus. The result of a Dockerfile build gives you precisely the reproducibility you desire. Every time you run a container from that image built from a Dockerfile, you'll get the same filesystem & environment.
Docker 1.6's "Content Addressable Image Identifiers" addresses your build in a year concern by allowing dockerfiles to refer to a digest to ensure you are building against exactly the image you expect (rather than the result of some build process that yum -y upgrades etc, which I think is what you were getting at).
The image (and thus container) are the same. Trying to rebuild it from scratch leaves it not the same, typically because the way people put together Dockerfiles and build images does not follow a standard. And it has to do with the container culture.
Let's take a Dockerfile for the CentOS 7 version of nginx (https://github.com/CentOS/CentOS-Dockerfiles/blob/master/ngi...)
Right off the bat I think: what the hell? Why are they doing a yum update? A yum update today may very well leave the system in a completely different state than a year ago. There's likely been some package updated in that time, which changes the state of the system. Right off the bat we're screwed. And what the hell is this?! There's no version, no build, no checksum. What the hell did we just install? If those packages change in a year, we're screwed. Not to mention 'epel-release' and 'tar' should be set as dependencies somewhere, and the type of dependencies too. Oh, cool, just use whatever the hell this config is which may or may not be different from the one that shipped with the original package. And let's just modify it for no apparent reason, too. And definitely make sure we have no way to know what version of that file we're using for this image. Lovely. The hell? We're pulling some random git sources from a git server which i'm willing to bet has no standard mirrors? On closer inspection it doesn't really look like a git server, but a host named git which hosts files whose names are a checksum, though we don't know where this is from or what exactly it refers to. If this server or file disappears, good luck knowing what in hell was being downloaded here.> Maven, ivy and sbt are the go-to tools for having your system download unsigned binary data from the internet and run it on your computer.
The root of the problem is that out of the total number of libraries available in language X, only a small subset is packaged in Debian/RHEL. This may be more egregious with large, Java enterprisy software, but you could easily end up with the same problem in Ruby or Python.
You cannot reasonably expect developers to package and maintain all their dependencies properly. The least worse solution would be to:
- still use maven to manage dependencies
- create a Debian/RHEL package incorporating the dependencies (effectively vendoring them in the package)
Unfortunately, it is not that simple, because you need to make sure that your vendored-in-the-package dependencies are somewhere where they will not conflict with another package with the same idea and the same dependencies (or better, the same idea and a different version of the same dependencies). Which means you need to keep them out of /usr/share/java and make sure the classpath points at the right location.
However, it seems that developer tend to avoid this kind of rigmarole and instead go for the "install dependencies as a local user" for certain classes of application (eg, webapps) because packaging is not fun.
I think that this is a good point, but it all comes down to quality control.
You wouldn't accept a new dependency into your project if it is buggy or has a bad API.
So why is bad packaging, a hacked-up build system or inability to build from an auditable source considered acceptable in many communities today?
The problem is that you normally cannot find a path to the developers through the web of trust, of course, but that's not Maven's fault. That's the fault of the web of trust (more accurately called "handful of strings of trust").
Debian/Red Hat code signing doesn't prove very much either. All it proves is that the package came from Debian or Red Hat. But did they modify the software along the way, doing some kind of MITM attack on the upstream developers? Quite possibly! At least with Maven you don't have that problem.
It is Maven's vault. Debian could have taken the same approach, but instead they have decided to vet prospective developers for a few years, and use one signing key for package indexes.
As a result, in Debian:
- It is feasible to actually check packages, because you don't need a large WoT.
- You know that the maintainer of a package was a least vetted for some time before they could put anything in the archive.
With Maven:
- It's infeasible to check the signatures, since nearly no one has a complete enough WoT.
- Anyone can submit a library, no vetting necessary.
- Maven does not check signatures by default.
So, basically the only thing you can do in Maven is hand-pick a small set of libraries. Verify them in some way and stick to specific versions, putting their checksums in your POM. Unfortunately, as the article already touches upon, too many Java libraries pull in half of the internet, so it may not be very practical.
Yes. But this creates a much higher barrier to entry. As usual, it is a trade-off, but it is obvious that with a Debian-like system, the number of packages on Maven/CPAN/Pypi/Rubygems would be considerably smaller.
(Of course it wouldn't provide any value, which is why no-one does it).
Why not? This is exactly what developers are expected to do. Every developer must manage dependencies for their application to work.
What you mean is they can't be reasonably expected to do it well.
In most situations, this is truly a trivial amount of investment relative to the overall cost of developing and maintaining an application.
There are ecosystems which make the task easier or more difficult/annoying, but that cost should be accounted for when choosing which development platform to use.
I wouldn't say that creating and owning packaging for, say 50 libraries (assuming you only deploy to a single platform) represents a "trivial amount of investment" for a standard small developer team.
I might reevaluate the suitability of that particular language for the project.
On a side, it only takes marginally more time to build a package for a distro as it does to build the package by itself. If you've already got the build done, 3 experienced guys could knock out 50 packages in a week. Inexperienced (in this task), but competent, devs should be able to do it in 2 weeks.
To my mind, that was what "devops" was supposed to be, but it's been a bit of a dogpile in the years since the term gained popularity.
Systems are opaque to most developers, and many developers wish to make their software opaque to the system on which it runs. This is a failure on behalf of our entire profession, not any one group.
Infrastructure software is in a bit of a renaissance period, but it's very early days. Packaging software is a total mystery to most developers. I don't even need to back that up with examples, most of us can recall the last time we can across a well packaged piece of software with joy due to sheer rarity. I'd be very surprised to find the average age of a Debian maintainer was trending anything but upwards, and steeply.
Containers are being misused, but that's because the alternatives we've been building for ourselves have not kept up with the strong user experience narrative of web and mobile software.
We need to do better.
In my mind, that is largely what devops is about - team ownership of the entire product, which includes infrastructure. Instead of having a silo'd 'ops' team writing ansible scripts and doing deployment, this should be part of the team (which could mean having an opsy guy on the team).
Anyways, as it pertains to containers, I think containers are more a practice than a principle. It tends to happen naturally when you want reproducible builds and continuous delivery. It's not really about making systems opaque to software, imo, but rather making your product artifacts reproducible (if you rely on running ./configure; make at deploy time, you never know what you'll end up with since dependencies are dynamically determined).
https://github.com/chef/devops-kungfu
https://www.youtube.com/watch?v=_DEToXsgrPc
Some idiots even advocating to have a whole snapshot of an OS attached to your "hello world", and even to make it what they call "purely functional" or even "monadic" (why not, of someone pays for that).
Unfortunately, there is no way to ignore complexity of versions and package dependencies or easily pushing it to "devops". Creating a zillion of "container images" with just your "reproducible development environment" or a whole "OS snapshot" just multiplies entities without a necessity.
Programmer must be aware of which version of what API implemented with what version of package or library he using and explicitly assert and maintain these requirements, like all the very few sane software projects (git, nginx, redis, postgress) do.
btw, the GNU autotools (which gives us ./configure) is somewhat evolved real-world solution - you have to explicitly check each version of each API both at the compile (build) time and refuse to build in case of unsatisfied dependencies and at the install time (and package manager must refuse to install if case of mismatch). This is the only way back to sanity, however "painful" it is.
Except that when you're doing anything that looks like an actual end-user application (as opposed to infrastructure), you end up using dozens of libraries which themselves have dependencies, so suddenly you're supposed to "explicitly assert and maintain" hundreds of different library versions, none of which is in any way relevant to the application you're building.
I myself see Docker containers as the only reasonable way for giving a service application to people to deploy on their machines, because even the programming runtime I need is 5 years out of date on Debian/Ubuntu, and installing that stuff manually is a) pain, and b) different on every operating system.
Take a look how git or nginx compiles from the source on any machine imaginable.
There is absolutely no fundamental problem with ./configure; make; make install.
> There is absolutely no fundamental problem with ./configure; make; make install.
The fundamental problem is incompatible versions of dependencies. Arguably it's in linux's dynamic linker rather than a problem with configure/make. But if you need to run something that depends on libfoo 2.3 and something else that depends on libfoo 2.4 on the same machine, you need something like docker.
The other approaches, such as Java's (where we "abstract out an OS") in practice leading only to a bigger mess, because it boils down to the very same libc, libm, libffi and friends. JVM is an ordinary userlevel program, so it obeys to the restrictions and rules for any other userlevel program. This is the sad truth for Java zealots.
Basically, one cannot ignore an OS (at least when you still want to dlopen cand call the stuff instead of re-implementing it poorly) - it is just a wrong idea, leading to all these ridiculous FS-inside-JVM implementations and other messed up layers of unnecessary, redundant abstractions.
System administration is still hard and it (necessity to think, understand and analyze) cannot be eliminated by some bunch of shell or ruby scripts and wishful thinking.