A note about nix: I've found the language to be very obtuse and the documentation out of date and fragmented. However the worst sin has been the attitude of several community members: insulting new developers, ignoring valid questions, and self-promotion at the expense of civility. I love the idea behind the system and like most of the community, but the very vocal jerks really put me off.
I've never learned the nix language, but that hasn't been an issue in my usage for the last couple years. It's really easy to add a new expression for a package, which is all I've ever used it for.
Sucks to hear about the negativity. For what it's worth, I've gotten a handful of pull requests accepted with positive experience each time.
I just wish to report that I had only good experience in communicating with Nix/Nixos/Nixpkgs community.
When someone could solve something that didn't work for me, they did. When no one could or didn't have time, they kept silent or politely tried to help me find the solution myself. And everything absolutely for free!
My mileage is different than yours. When I started using NixOS on my laptop I've asked for help a bunch of times on the #nixos IRC channel and people were really helpful.
This was somewhere in the second half of last year.
Both nix and guix seem to have potential. Not sure if they are the "silver-bullet" (in the good sense of actually nailing a hard problem well) but perhaps.
IMO they are a bit too (local) package oriented than necessary.
Nix + Nixops gives him everything he wants. He'd have to learn to write nix expressions, which does put people off.
Containers are great if you've got racks full of actual, physical servers and want to use them efficiently. Most people don't have that problem, though.
If you’re deploying C programs that rely on system
libraries, things may get tricky if you cross flavors or
versions of linux. But you can probably deploy static-linked
executables more easily than setting up docker.
For various reasons static linking isn't a good idea for C applications. glibc doesn't work well when statically linked. musl works well. But glibc has strong backward compatibility using symbol versioning. It's not usually worth the trouble to build and link aginst musl if your distro uses glibc. What you should be worried about are all the other third-party libraries, which aren't often written to be statically linked--because of namespace pollution, because of slightly different semantics between dynamic and static code, especially at application startup.
Fortunately, Linux supports packaging apps much like macOS bundles, where all shared objects are kept under a relocatable subtree. When building a binary that will be installed in bin/, for example, just use the following GCC flags:
The first flag tells the linker to find libraries in the lib/ directory adjacent to the binary file itself. The second flag tells the linker to try LD_LIBRARY_PATH first rather than the embedded rpath. Debugging and regression testing can be very difficult without the ability to use LD_LIBRARY_PATH, and unfortunately LD_LIBRARY_PATH has lower precedence than embedded rpaths, thus the need for --enable-new-dtags to change the default behavior.
Note that $ORIGIN is a special string that is expanded by the runtime linker, not by the shell at compile time. It's an unfortunate choice of syntax. Getting $ORIGIN to pass through to the compiler without being evaluated by a shell can be tricky when dealing with recursive make and shell invocations, such as from an RPM spec file.
Another example: when building a Lua module I'll use
-Wl,-rpath,'$ORIGIN/../..' -Wl,--enable-new-dtags
because Lua modules are usually installed under $(prefix)/lib/lua/5.?/, two directories below lib/.
And then you might just end up using setcap on it anyhow, perhaps to load privileged ports, which is treated like suid, making it so that the linker won't look at LD_LIBRARY_PATH, in which case you have to create a conf file with your paths and feed it to ldconfig after using ldd on the program because you're confused about why it suddenly doesn't load its libraries.
And there's another fun security restriction in there where chown erases any setcap permissions you had applied, just in case you had to adjust something and didn't understand why you suddenly lost your setcap permissions.
Why? If you control that, you likely also control PATH and others. There are lots of variables which mean game over if you have write access to them. And you also need a local fs write access to use it, so that's another step. I know it could be something to restrict, but it's really far down the list... (configuring selinux/apparmor would solve the same issue but in a more generic way)
Using -Wl,--enable-new-dtags merely restores the default behavior. More specifically, it only restores the usual precedence behavior.
Most people don't set the DT_RPATH ELF tag. It's usually considered best practice to NOT set DT_RPATH. And that's generally good advice, except that advice assumes absolute paths. Most people aren't familiar with $ORIGIN, which allows using relative rpaths. In any event, my point is that most environments already allow LD_LIBRARY_PATH. If it's been restricted then --enable-new-dtags won't change that.
What --enable-new-dtags actually does is set DT_RUNPATH instead of DT_RPATH. DT_RPATH has higher precedence than LD_LIBRARY_PATH; DT_RUNPATH has lower precedence.
To reiterate, DT_RPATH and DT_RUNPATH are independent of LD_LIBRARY_PATH, and don't effect whether it's permitted or not. I recommend --enable-new-dtags to avoid confusion and unintentional side-effects.
I've known about $ORIGIN for awhile, but only recently hit upon --enable-new-dtags. I was hacking on a project recently ported to and packaged for CentOS 7. The relevant components were a Lua C module (acme.foo) binding a shared library (libacme-foo). The in-tree unit and regression tests used LUA_PATH, LUA_CPATH, and LD_LIBRARY_PATH so the in-tree objects were loaded instead of any that might happen to be installed on the system. I was adding some debugging code to the shared library and running `make check`, but my changes didn't seem to be coming through in the tests. I was totally confused.
After far too much wasted time, I eventually realized that CentOS 7's base Lua 5.1 package was built with an embedded rpath of /usr/lib64. A version of the package I was hacking on was already installed on the system. So /usr/lib64/libacme-foo.so was linked when dlopen'ing the Lua C module instead of the in-tree libacme-foo.so that should have been found with LD_LIBRARY_PATH. That's how I became familiar with the precedence of rpath and LD_LIBRARY_PATH, and the availability of the --enable-new-dtags option.
I'm not sure why Red Hat built their core Lua 5.1 binary with an rpath, but they almost certainly should have used --enable-new-dtags when doing so. Notably, /usr/bin/perl on CentOS 7 is built with --enable-new-dtags; it has DT_RUNPATH=/usr/lib64/perl5/CORE. Just as notably, AFAICT Python isn't built with any rpath.
Habitat.sh seems to solve all your concerns. It's a clean room build tool that runs a supervisor on pid 1 of your containers and enables them to be configable, plus lots of other goodness that answers "how do we manage containers after day 1" style problems.
Yes, the point of Convox is that it deploys your docker containers to AWS using your docker-compose config. It sets up a VPC for you, ELBs, autoscaling group, private docker repository, the whole deal :)
* config file for resources.
* It does log capture/delivery mostly.
* it does chroot exec.
* it handles network ports, file paths, etc
* handles batch workloads.
It can't handle the load balancer piece directly, but if you use the consul tie-in, you can make it work, since it does service discovery, and then something like fabio [1] /traefik can handle load-balancing.
In the long term I predict that base OS everywhere will improve support for deployment, workload scheduling, resource allocation, endpoint discovery, and dependency management. These will match and eventually surpass the additional capabilities that containers offer, and then we can all go back to putting files on a server and restarting a process, which is all that 99% of us actually need.
The alternative is that container platforms grow their own hypervisor, effectively replace operating systems, and then build out their capabilities in storage, filesystems, networking etc until they are functionally basically indistinguishable from operating systems c.f. VMware. But I think the former option is more likely, it's just a gut feel.
Docker containers can still support hot reloading of Erlang/Elixir applications. It's just not a particularly-pleasant experience OOTB. It's almost always easier to just restart the container or to not use containers in the first place.
Config files as copied around by scp and edited by hand are a pita -- but as checked into source control as a template and staged to machines by your deploy; the very simplest and most robust solution.
Yeah man, I could setup a quorum of consul boxes, integrate consul's api into my application, figure out how to manage changes in that system, and then do as you say -- make a change management system to apply deltas on deploy.
Or, I could put one more file on each box in addition to the other 100,000 files I put on there...
Obviously you wouldn't choose consul for this one feature, but if you're already relying on consul as service discovery in your network, why not?
You don't need to invent a change management system, you just need to utilize from a myriad of existing consul OSS.
I'm not saying consul makes all the sense in the world for you, but there's definitely sound architectural thinking behind what the parent is talking about. It's not just a bunch of crazy talk :-D
Can consul create change log trails to keep the auditors happy? Some of us are working in environments that require more rigor than a pet project.
Not throwing stones, a serious question. Made a few decisions to favor log files and change control systems vs a web app that had ACL's but no real auditing or change control mechanisms.
Just because you can store configuration files in consul doesn't mean you should. Consul's KV is strongly consistent, you don't need this kind of guarantees for configuration, and you are inheriting drawbacks caused by the overhead.
As anything it will work fine for small setups, but it will start having issues once you will add more nodes.
Depending on your goal using VCS (for example svn, yes not as hip, but it has a smaller footprint of not cloning the repo) provides versioning, or a database (which provides easy way of changing values) or a hybrid approach will be better.
Consul has some nice features around change notification that make features like pushing a change in conf without needing a code deploy trivial to implement. Also consul servers are strongly consistent but agents are eventually consistent, fwiw, so that statement will mean different things depending on how you've set up your infrastructure.
There's also nothing to stop you from pushing config into git and then having a build that just syncs files to consul kv.
That feature is hacked around http, where the client is constantly connected over http (through long standing connections) and reconnects every 5 minutes (by default) if there was a change. You would be much better to use Zookeeper for this (through watches) even though Zookeeper is also not the best tool for this for similar reasons.
Using a distributed locking system for notification is really missing why it is for, why not use Postgres' notifications, AMQP is it because it's easier to tell it is not the right tool for the job?
I currently have colleagues implementing an event bus (although the use cases they mentioned is that they will most of the time send messages to individual machines or small group of them) for over 10k machines. Their first choice was to use Consul's KV, but after persuading them that we potentially could have scaling issues, they decided to drop it, unfortunately they are still misguided and plan to use Serf which will have its own issues as well, this time it might be network capacity.
The issue though is much simpler. We are all communicating through IP protocol, which is inherently point to point, why building abstractions to the problem, when network already matches perfectly the requirements. I think we are so used to using a tool for everything that it feels weird to not have to use one. I was able to have a working POC in 10 lines of Python code (of course a final solution would be more). Argument against using what I proposed was to not implement custom solutions because of maintenance issues (what will happen if I decide to quit), the thing is that one will write far more code to implement the solution with Serf or Consul.
> Also consul servers are strongly consistent but agents are eventually consistent, fwiw, so that statement will mean different things depending on how you've set up your infrastructure.
This is not true, KV is strongly consistent including agents (otherwise there's no point in being strongly consistent since all communication supposed to happen through agents), you could relax requirements on individual requests by adding ?consistency=stale, but if your application makes all requests this way, then it is a sure sign that you chose a wrong tool for the job.
> There's also nothing to stop you from pushing config into git and then having a build that just syncs files to consul kv.
Similarly there's nothing stopping you from using Solr as NoSQL storage, encode data using thrift and then store it in 2 column table (id and blob) in Postgres (then have a tool to edit the data directly which encodes/decodes the thrift data), using Coherence (Oracle's P2P caching solution) as primary source of data that is ETLed from Postgres data[1] etc. All those things I have seen done and it is really painful to watch.
[1] Perhaps this doesn't look as bad, but ETL to coherence took 5 hours, and Coherence since it was a caching solution if it detected a network partition just to be safe it decided to invalidate all data it had. The solution was to throw money at a problem, so why not have 2 or 3 environments with initial setup so while something happens to one you can switch to another while reloading the data. This kind of worked until a contractor installing a security system in a data center decided to plug in a dumb switch to two main distribution switches configured to work in HA, creating a loop and shutting entire network down. The site was down for hours until all of the data was loaded back.
> What will it take to build this? 6 months x 3 smart devs who understand the linux kernel. So like $600k.
It seems like PaaSes are the new JavaScript frameworks[0]. Kubernetes now has most of the "critical pieces" mentioned and Helm addresses the "rigid" config criticism. It's not newbie proof yet but I doubt your 1.5 man-year project would get anywhere close to K8s is in terms of maturity and stability. Please do your homework and consider contributing to an existing project before re-inventing the wheel.
This exactly hits the problem with containers (and especially Docker and Kubernetes, but to a lesser extent CoreOS's model as well). The crows that seems to like/use it seem to be the people that aren't responsible for maintenance, security etc. and may not want to become the responsible party either. It's often developers that are either simply a lone wolf or part of a team that is big enough that release management falls on one or two people and 'the rest' just wants to churn out code in 'their way' and have that 'way' magically work in production too. Sadly, that is not how the world works.
You will always need some form of configuration management, release management, infrastructure and operations, no matter how compartmentalised and 'serverless' the software coders think they are. Right now, the idea seems to be that if you stick your code and practically a complete user land in a container, you don't have to do those things, as if old style deployments were the reason for those practises to be in place. Obviously, it is the other way around, and by trying to bypass them, you will run into the exact problems they are meant to prevent. (i.e. not knowing what is available/what is released, not knowing the configuration within the infrastructure in a global scale, not knowing the side-effects of two opposing configurations because you have no management for that, not knowing what versions are used because it is spread out over different/many unmanaged files inside containers somewhere, not knowing what is secure/patched, not knowing what else is influencing your environment, no control over parameters that should be globally overridden...)
We take a different approach: you get do write your software, and may only configure requirements within its context. Those have to be defined in a SaltStack Formula with sane defaults, that has to run in a local environment, but also DTAP-style chains of environments with no changes to the app. You don't get a say in what it will run on, but your required facilities will be available. If it doesn't pass automated testing (i.e. sticking it in a VM, running a highstate and expecting a working app) it doesn't get passed to DevOps and you're stuck until you fixed your stuff. With regards to scaling, we make sure that a few calls are always available to the app and/or it's formula, so that we can register where it is and connect it to the correct balancing pools (i.e. incoming requests, but also FS, DB, queue pools). When it's up, it registers, and as long as the automatic health checks pass it gets requests sent to it. If the load is too high, we scale up, if it's been idle too long it scales down. But the app/software/code isn't aware of that nor does it need to be, and using the reactor system, we don't need 'extra' management software to make that happen. You set your KPI's and health checks where you need them, just like you would on any other setup, and you're golden.
Operational platforms have been built on containers and more are arriving all the time. Red Hat rebuilt OpenShift on Docker and Kubernetes. I work on Cloud Foundry for Pivotal. There are many others, including self-developed platforms like the one you've built.
The general platforms are written to make it easy for operators and developers to remain friends, by sharply drawing the boundaries of who does what. Operations gets manageable black boxes and development gets free platform features (routing, logging, service injection, balancing, healthchecks etc).
If a developer needs to care about how these are implemented, or do work above baseline to get these features, then my peers and I have more work to do.
Your last paragraph could clearly be implemented in k8s / etc. but the defining characteristic is that you've built organizational controls to protect & optimize a distributed system. And the main problems you bemoan in the first paragraph are issues of, again, organizational control & maturity.
It's almost, like, technologies are just tools that can be either applied or misapapplied to a given domain, either for better or for worse. Huh.
It's obviously all about implementation and application, but instead of always reading 'Docker is the solution' it would be nice if it were listed as 'just another tool in your toolbox'. The problem I personally have with this and some other 'emerging' technologies is that they are touted as a 'replacement' for Ops, management, or infrastructure, while in practise, it's obviously not.
People like the idea of isolation that containers (LXC) provide. We don't have to worry about stepping on toes, my container works with any number of other containers running on the same host, not sometimes, all times.
The env is moving fast, Kubernetes and Helm are on a tear. It's also worth taking another look at cluster standup CoreOS+Vagrant/Tectonic/Minikube/kubeadm are filling in gaps.
minikube was not quite primetime when I tried it, but I saw release notes a few weeks later that fixed everything I ran into. (IIRC it was ingress or DNS problems).
Your impressions about Kubernetes sounded quite out of date when I read your post. I strongly suggest you give things another try. Things move fairly fast in this area.
Yep with today's minikube, you can do ingress. For example, I'll point myproject.minikube.local to 192.168.99.100 and use the stable/nginx-ingress helm chart.
When I'm running my application stack on minikube, I can set a helm template variable to myproject.minikube.local.
The nginx ingress controller will route the myproject traffic to the proper backing service.
On prod, I'll set the template variable to super-social-money-maker.com and turn up the stack with a helm install. Here, ELB does the ingress for me, but as an app developer, I don't have to be concerned with the particulars of the cluster's ingress implementation.
The VM overhead is ludicrous compared to LXC containers though, even with all that pass-through that modern hypervisors do. I don't get all the hate for containers from the VM folk, those technologies are by no means exclusive, it's just that spinning up an entire virtual machine to run some microservice strikes me as wasteful.
I think your confusion is based on the assumption of microservices. If you're deploying something which shuffles packets, yes, containers are great for quickly spawning instances and getting more utilization on a given host. Once you have apps which use significant resources and persist non-trivial amounts of data – i.e. what most developers work on – the difference becomes a lot less significant. (There's a similar point in overhead: unless you're doing privileged calls in a loop, hypervisor overhead hasn't been significant for at least a decade unless you're talking about something like VirtualBox)
Note that I'm not saying containers are bad, just that the people who disagree are almost certainly speaking from experience in different environments where other trade offs make sense compared to where you're working.
> Once you have apps which use significant resources and persist non-trivial amounts of data – i.e. what most developers work on – the difference becomes a lot less significant
If you spin up the VM once and it runs for days/weeks that is true. In a cluster scheduler environment you usually have immutable images where every change requires replacement of the workload. In such scenario your VM or container might get replaced a few time a day, during every deploy. That's where VM startup time becomes an issue.
> In a cluster scheduler environment you usually have immutable images where every change requires replacement of the workload. In such scenario your VM or container might get replaced a few time a day, during every deploy.
That's kind of my point: that's a relatively unusual workload and even that assumes that you have both insignificant amounts of persistent data and frequent new deployments. If you're doing anything which copies much data or does non-trivial initialization work, saving a minute by launching a container vs. a new VM is still a healthy improvement but it's not the expensive part of a new deployment and unless you're churning many jobs per hour it's a rounding error in the total execution time.
(I'd still prefer a container for that kind of workload but it'd be for the consistency & simplicity aspects)
> that's a relatively unusual workload and even that assumes that you have both insignificant amounts of persistent data and frequent new deployments.
That might be true for enterprise (I simply don't know) but in the startup world these are these are actually the most common workloads: Stateless apps that scale vertically.
I agree that startup time doesn't matter much if you need to do lot of initializations. But in my experience in most infrastructures these are a few databases of which there are orders of magnitude less than other workloads.
> my iterative dev setup often isn’t that similar to my ‘under docker’ setup
This seems like a strange complaint to me, because it's one of the best things about iterating with docker. Make a change, do a build (super quick because of fs layers) then run it. If it works, ship it!
Not only that, but now I can do the same thing in Windows. It's a wonderful time to develop linux applications.
I don't know about the OP, but a lot of
people rely on hot code reloading to see live changes, be it a Node.js app or a React UI. The Docker build workflow is way too slow for that. When I work on React apps, I expect the UI to seamlessly reload every time I save a file.
You can accomplish this perfectly well with Docker, of course, though you have to use volumes and so on. Docker for Mac's mounting support has inotify/fsevents on both sides, though the volumes are not fast. (Fast enough for a React app, but enough for a Postgres database.)
Exactly. First thing I do when setting up a new project now is to set up a Dockerfile for the container I will build things in. For scripting languages etc. it's also easy enough to just mount your working directory anyway, so the Docker container provides the environment but you don't need to rebuild the container for every change.
E.g. for my Ruby web projects I'll often have things like this in a Makefile:
With a Dockerfile that might be as simple as this:
FROM ruby:2.3.1-onbuild
EXPOSE 80
CMD bundle exec puma --port 80
Gives me a predictable environment where I "only" rebuild the Docker container when I add/remove dependencies, and when I run it in development, shotgun will hot-reload the Ruby code from the mounted host directory.
Saves me from polluting my hosts with whatever version of code I need for different projects.
At this point there's very little code I'm willing to run outside of containers, not for security reasons but because they prevent my laptop from getting full of cruft.
I couldn't imagine going back to developing outside of containers.
rkt seems to meet his "define resource access in a config file" requirement in principle.
The separation in rkt between setting up the isolation and handling the images or running of the code means that you can opt to launch a rkt image in anything from an actual chroot to a full container or even a full VM, or you can write your own if you have weird needs (e.g. if you want to do a seccomp-bpf type sandbox, you can do that).
E.g. I have deployments with etcd in a chroot (using rkt), and the rest deployed as containers, which with rkt defaults to use systemd-nspawn, and which means they have systemd running as the init in the container too and logging goes straight to the journal just like everything else.
Cron really depends on scale. For small deployments it's simple enough to simply use a cron or systemd and use a simple wrapper that uses e.g. Etcd or Consul or even a database table to arbitrate who gets to run for jobs that shouldn't run more than once. In practice depending on exactly what you do it's often simpler/better to write the jobs so it doesn't matter if they're started more than once anyway. Seen too many nasty surprises in systems where people just assumed nobody would ever end up with the scripts overlapping in time only for something to slow down processing until they do just that anyway. If you're first doing that, all you need to do to handle a smaller distributed system is to put your locking mechanism somewhere shared.
At scale it becomes more complex, but the overall principle still holds, though you may want to separate scheduling aka. the various distributed cron's.
As for load balancer, I agree it needs standardising, but it really is a small enough issue for smaller setups that it's fairly easy to solve. I've come to rely on setups that uses Nginx and rewrites incoming hostnames to a local hostname (e.g. foo.com => foo.com.service) and use resolvers in Nginx to point them to a SkyDNS instance. That means all the container config needs to do is set/delete Etcd keys to receive traffic for a given hostname.
But there's several load balancers out there that now can pull config directly from a suitable shared datastore, be it Redis or Etcd etc.
> When I used kube in 2016 the cluster turnup support was bad; it seemed like you could use it managed in the G cloud but not anywhere else.
You might like to look at Kubo[1], which was jointly built by Googlers and Pivots to make this more manageable. We built it based on requests from joint customers who want BOSH as their infrastructure-level platform and the choice of Cloud Foundry or Kubernetes for application-level platform.
So many docker containers I use at work just have a ton of apt-get or yum statements (depending on a debian or centos base) and people rarely pin dependency versions. I really feel for the "repeatable build" issue, not to mention dependency rot.
If you want to keep things up to date, there are many examples scripts out there that either 1) checking to see if your base container is out of date (if someone updated jessie:8, you should rebuild) and 2) checking if any packages are out of date (sometimes limiting to just security updates) meaning you run apk, yum, apt, etc. within each container (or extract their package lists and examine them outside the container).
You gain a lot in the isolation, but debugging can become more difficult and you've introduced new areas of dependency rot.
I generally like containers, but hate how there are so many different orchestration formats (k8s, marathon, nomad, swarm, etc.) many different network layers (weave, flannel, etc.) and many different ways to package/fit components together (CoreOS, DC/OS, etc.)
I'd like to build something that attaches cryptocurrency payments to API calls, basically a 402 payment required implemented with the Lightning Network.
With such a system in place, pulling updates could be pinned to a particular API call, with a particular payment address, which itself was attached to a particular data store. In theory, as long as you had access to the wallet that did the build before, you'd be able to do the same build at a later date.
I'm greatly simplifying this here, but my conclusion after getting a crypto payment OpenStack instance launcher PoC working was that applying crypto payments to deployment methods basically provided a lightweight federated layer that allowed logically linking processes, similar to what you would get with a Zapier/Apigee/IFTTT knockoff. Applied to microservice architectures it would allow deployments to be both trustworthy and repeatable at later dates.
190 comments
[ 2.1 ms ] story [ 274 ms ] threadIsn't load balancer standardization found with Kubernetes Ingress? Though it doesn't let you select the lb algorithm though.
Isn't cron found in Kubernetes Cron Jobs?
Can you post a link to how nix handles resource permissioning?
Sucks to hear about the negativity. For what it's worth, I've gotten a handful of pull requests accepted with positive experience each time.
When someone could solve something that didn't work for me, they did. When no one could or didn't have time, they kept silent or politely tried to help me find the solution myself. And everything absolutely for free!
This was somewhere in the second half of last year.
Containers are great if you've got racks full of actual, physical servers and want to use them efficiently. Most people don't have that problem, though.
Fortunately, Linux supports packaging apps much like macOS bundles, where all shared objects are kept under a relocatable subtree. When building a binary that will be installed in bin/, for example, just use the following GCC flags:
The first flag tells the linker to find libraries in the lib/ directory adjacent to the binary file itself. The second flag tells the linker to try LD_LIBRARY_PATH first rather than the embedded rpath. Debugging and regression testing can be very difficult without the ability to use LD_LIBRARY_PATH, and unfortunately LD_LIBRARY_PATH has lower precedence than embedded rpaths, thus the need for --enable-new-dtags to change the default behavior.Note that $ORIGIN is a special string that is expanded by the runtime linker, not by the shell at compile time. It's an unfortunate choice of syntax. Getting $ORIGIN to pass through to the compiler without being evaluated by a shell can be tricky when dealing with recursive make and shell invocations, such as from an RPM spec file.
Another example: when building a Lua module I'll use
because Lua modules are usually installed under $(prefix)/lib/lua/5.?/, two directories below lib/.And there's another fun security restriction in there where chown erases any setcap permissions you had applied, just in case you had to adjust something and didn't understand why you suddenly lost your setcap permissions.
Most people don't set the DT_RPATH ELF tag. It's usually considered best practice to NOT set DT_RPATH. And that's generally good advice, except that advice assumes absolute paths. Most people aren't familiar with $ORIGIN, which allows using relative rpaths. In any event, my point is that most environments already allow LD_LIBRARY_PATH. If it's been restricted then --enable-new-dtags won't change that.
What --enable-new-dtags actually does is set DT_RUNPATH instead of DT_RPATH. DT_RPATH has higher precedence than LD_LIBRARY_PATH; DT_RUNPATH has lower precedence.
To reiterate, DT_RPATH and DT_RUNPATH are independent of LD_LIBRARY_PATH, and don't effect whether it's permitted or not. I recommend --enable-new-dtags to avoid confusion and unintentional side-effects.
I've known about $ORIGIN for awhile, but only recently hit upon --enable-new-dtags. I was hacking on a project recently ported to and packaged for CentOS 7. The relevant components were a Lua C module (acme.foo) binding a shared library (libacme-foo). The in-tree unit and regression tests used LUA_PATH, LUA_CPATH, and LD_LIBRARY_PATH so the in-tree objects were loaded instead of any that might happen to be installed on the system. I was adding some debugging code to the shared library and running `make check`, but my changes didn't seem to be coming through in the tests. I was totally confused.
After far too much wasted time, I eventually realized that CentOS 7's base Lua 5.1 package was built with an embedded rpath of /usr/lib64. A version of the package I was hacking on was already installed on the system. So /usr/lib64/libacme-foo.so was linked when dlopen'ing the Lua C module instead of the in-tree libacme-foo.so that should have been found with LD_LIBRARY_PATH. That's how I became familiar with the precedence of rpath and LD_LIBRARY_PATH, and the availability of the --enable-new-dtags option.
I'm not sure why Red Hat built their core Lua 5.1 binary with an rpath, but they almost certainly should have used --enable-new-dtags when doing so. Notably, /usr/bin/perl on CentOS 7 is built with --enable-new-dtags; it has DT_RUNPATH=/usr/lib64/perl5/CORE. Just as notably, AFAICT Python isn't built with any rpath.
You might argue that containers are fancy chroots. They are just built from scratch and packed each time
Show me that $400k/year linux kernel job.
150-200k in salary + 50-100k in stock + 50k-100k for health/401k/parentalleave/food//office/hardware/concierge/gym + employer taxes + ...
[0] https://www.nomadproject.io/ [1] https://github.com/fabiolb/fabio [2] https://docs.traefik.io/
If you develop in Elixir or Erlang, there's an additional issue that containers (such as docker) tend to interfere with hot code reload.
There are work-in-progress solutions (such as Elixir Distillery) that emphasize turning your app into a single package which can be deployed anywhere.
Even if you don't use Elixir, you may enjoy seeing how the Distillery packaging system works, and how hot code reload works.
A good intro is http://www.east5th.co/blog/2016/12/26/deploying-elixir-appli...
In the long term I predict that base OS everywhere will improve support for deployment, workload scheduling, resource allocation, endpoint discovery, and dependency management. These will match and eventually surpass the additional capabilities that containers offer, and then we can all go back to putting files on a server and restarting a process, which is all that 99% of us actually need.
The alternative is that container platforms grow their own hypervisor, effectively replace operating systems, and then build out their capabilities in storage, filesystems, networking etc until they are functionally basically indistinguishable from operating systems c.f. VMware. But I think the former option is more likely, it's just a gut feel.
After using Consul, I never want to go back to config files again.
https://www.consul.io/intro/
Or, I could put one more file on each box in addition to the other 100,000 files I put on there...
You don't need to invent a change management system, you just need to utilize from a myriad of existing consul OSS.
I'm not saying consul makes all the sense in the world for you, but there's definitely sound architectural thinking behind what the parent is talking about. It's not just a bunch of crazy talk :-D
Not throwing stones, a serious question. Made a few decisions to favor log files and change control systems vs a web app that had ACL's but no real auditing or change control mechanisms.
As anything it will work fine for small setups, but it will start having issues once you will add more nodes.
Depending on your goal using VCS (for example svn, yes not as hip, but it has a smaller footprint of not cloning the repo) provides versioning, or a database (which provides easy way of changing values) or a hybrid approach will be better.
There's also nothing to stop you from pushing config into git and then having a build that just syncs files to consul kv.
Using a distributed locking system for notification is really missing why it is for, why not use Postgres' notifications, AMQP is it because it's easier to tell it is not the right tool for the job?
I currently have colleagues implementing an event bus (although the use cases they mentioned is that they will most of the time send messages to individual machines or small group of them) for over 10k machines. Their first choice was to use Consul's KV, but after persuading them that we potentially could have scaling issues, they decided to drop it, unfortunately they are still misguided and plan to use Serf which will have its own issues as well, this time it might be network capacity.
The issue though is much simpler. We are all communicating through IP protocol, which is inherently point to point, why building abstractions to the problem, when network already matches perfectly the requirements. I think we are so used to using a tool for everything that it feels weird to not have to use one. I was able to have a working POC in 10 lines of Python code (of course a final solution would be more). Argument against using what I proposed was to not implement custom solutions because of maintenance issues (what will happen if I decide to quit), the thing is that one will write far more code to implement the solution with Serf or Consul.
> Also consul servers are strongly consistent but agents are eventually consistent, fwiw, so that statement will mean different things depending on how you've set up your infrastructure.
This is not true, KV is strongly consistent including agents (otherwise there's no point in being strongly consistent since all communication supposed to happen through agents), you could relax requirements on individual requests by adding ?consistency=stale, but if your application makes all requests this way, then it is a sure sign that you chose a wrong tool for the job.
> There's also nothing to stop you from pushing config into git and then having a build that just syncs files to consul kv.
Similarly there's nothing stopping you from using Solr as NoSQL storage, encode data using thrift and then store it in 2 column table (id and blob) in Postgres (then have a tool to edit the data directly which encodes/decodes the thrift data), using Coherence (Oracle's P2P caching solution) as primary source of data that is ETLed from Postgres data[1] etc. All those things I have seen done and it is really painful to watch.
[1] Perhaps this doesn't look as bad, but ETL to coherence took 5 hours, and Coherence since it was a caching solution if it detected a network partition just to be safe it decided to invalidate all data it had. The solution was to throw money at a problem, so why not have 2 or 3 environments with initial setup so while something happens to one you can switch to another while reloading the data. This kind of worked until a contractor installing a security system in a data center decided to plug in a dumb switch to two main distribution switches configured to work in HA, creating a loop and shutting entire network down. The site was down for hours until all of the data was loaded back.
It seems like PaaSes are the new JavaScript frameworks[0]. Kubernetes now has most of the "critical pieces" mentioned and Helm addresses the "rigid" config criticism. It's not newbie proof yet but I doubt your 1.5 man-year project would get anywhere close to K8s is in terms of maturity and stability. Please do your homework and consider contributing to an existing project before re-inventing the wheel.
[0] https://github.com/search?q=paas
You will always need some form of configuration management, release management, infrastructure and operations, no matter how compartmentalised and 'serverless' the software coders think they are. Right now, the idea seems to be that if you stick your code and practically a complete user land in a container, you don't have to do those things, as if old style deployments were the reason for those practises to be in place. Obviously, it is the other way around, and by trying to bypass them, you will run into the exact problems they are meant to prevent. (i.e. not knowing what is available/what is released, not knowing the configuration within the infrastructure in a global scale, not knowing the side-effects of two opposing configurations because you have no management for that, not knowing what versions are used because it is spread out over different/many unmanaged files inside containers somewhere, not knowing what is secure/patched, not knowing what else is influencing your environment, no control over parameters that should be globally overridden...)
We take a different approach: you get do write your software, and may only configure requirements within its context. Those have to be defined in a SaltStack Formula with sane defaults, that has to run in a local environment, but also DTAP-style chains of environments with no changes to the app. You don't get a say in what it will run on, but your required facilities will be available. If it doesn't pass automated testing (i.e. sticking it in a VM, running a highstate and expecting a working app) it doesn't get passed to DevOps and you're stuck until you fixed your stuff. With regards to scaling, we make sure that a few calls are always available to the app and/or it's formula, so that we can register where it is and connect it to the correct balancing pools (i.e. incoming requests, but also FS, DB, queue pools). When it's up, it registers, and as long as the automatic health checks pass it gets requests sent to it. If the load is too high, we scale up, if it's been idle too long it scales down. But the app/software/code isn't aware of that nor does it need to be, and using the reactor system, we don't need 'extra' management software to make that happen. You set your KPI's and health checks where you need them, just like you would on any other setup, and you're golden.
Operational platforms have been built on containers and more are arriving all the time. Red Hat rebuilt OpenShift on Docker and Kubernetes. I work on Cloud Foundry for Pivotal. There are many others, including self-developed platforms like the one you've built.
The general platforms are written to make it easy for operators and developers to remain friends, by sharply drawing the boundaries of who does what. Operations gets manageable black boxes and development gets free platform features (routing, logging, service injection, balancing, healthchecks etc).
If a developer needs to care about how these are implemented, or do work above baseline to get these features, then my peers and I have more work to do.
It's almost, like, technologies are just tools that can be either applied or misapapplied to a given domain, either for better or for worse. Huh.
The env is moving fast, Kubernetes and Helm are on a tear. It's also worth taking another look at cluster standup CoreOS+Vagrant/Tectonic/Minikube/kubeadm are filling in gaps.
When I'm running my application stack on minikube, I can set a helm template variable to myproject.minikube.local. The nginx ingress controller will route the myproject traffic to the proper backing service.
On prod, I'll set the template variable to super-social-money-maker.com and turn up the stack with a helm install. Here, ELB does the ingress for me, but as an app developer, I don't have to be concerned with the particulars of the cluster's ingress implementation.
When you run a bigger site, a lot services are using entire boxes/VMs so it doesn't make sense to stack them.
When you you have tons of systems, you optimize for easier management and debugging. You do not care about "wasting" some resources.
Note that I'm not saying containers are bad, just that the people who disagree are almost certainly speaking from experience in different environments where other trade offs make sense compared to where you're working.
If you spin up the VM once and it runs for days/weeks that is true. In a cluster scheduler environment you usually have immutable images where every change requires replacement of the workload. In such scenario your VM or container might get replaced a few time a day, during every deploy. That's where VM startup time becomes an issue.
That's kind of my point: that's a relatively unusual workload and even that assumes that you have both insignificant amounts of persistent data and frequent new deployments. If you're doing anything which copies much data or does non-trivial initialization work, saving a minute by launching a container vs. a new VM is still a healthy improvement but it's not the expensive part of a new deployment and unless you're churning many jobs per hour it's a rounding error in the total execution time.
(I'd still prefer a container for that kind of workload but it'd be for the consistency & simplicity aspects)
That might be true for enterprise (I simply don't know) but in the startup world these are these are actually the most common workloads: Stateless apps that scale vertically.
I agree that startup time doesn't matter much if you need to do lot of initializations. But in my experience in most infrastructures these are a few databases of which there are orders of magnitude less than other workloads.
This seems like a strange complaint to me, because it's one of the best things about iterating with docker. Make a change, do a build (super quick because of fs layers) then run it. If it works, ship it!
Not only that, but now I can do the same thing in Windows. It's a wonderful time to develop linux applications.
You can accomplish this perfectly well with Docker, of course, though you have to use volumes and so on. Docker for Mac's mounting support has inotify/fsevents on both sides, though the volumes are not fast. (Fast enough for a React app, but enough for a Postgres database.)
E.g. for my Ruby web projects I'll often have things like this in a Makefile:
With a Dockerfile that might be as simple as this: Gives me a predictable environment where I "only" rebuild the Docker container when I add/remove dependencies, and when I run it in development, shotgun will hot-reload the Ruby code from the mounted host directory.Saves me from polluting my hosts with whatever version of code I need for different projects.
At this point there's very little code I'm willing to run outside of containers, not for security reasons but because they prevent my laptop from getting full of cruft.
I couldn't imagine going back to developing outside of containers.
EDIT: spelling, sorry bears, no metal for you.
You know, the entire time I keep telling my self "don't type bear, don't type bear" and my wife distracted me and I typed bear.
The separation in rkt between setting up the isolation and handling the images or running of the code means that you can opt to launch a rkt image in anything from an actual chroot to a full container or even a full VM, or you can write your own if you have weird needs (e.g. if you want to do a seccomp-bpf type sandbox, you can do that).
E.g. I have deployments with etcd in a chroot (using rkt), and the rest deployed as containers, which with rkt defaults to use systemd-nspawn, and which means they have systemd running as the init in the container too and logging goes straight to the journal just like everything else.
Cron really depends on scale. For small deployments it's simple enough to simply use a cron or systemd and use a simple wrapper that uses e.g. Etcd or Consul or even a database table to arbitrate who gets to run for jobs that shouldn't run more than once. In practice depending on exactly what you do it's often simpler/better to write the jobs so it doesn't matter if they're started more than once anyway. Seen too many nasty surprises in systems where people just assumed nobody would ever end up with the scripts overlapping in time only for something to slow down processing until they do just that anyway. If you're first doing that, all you need to do to handle a smaller distributed system is to put your locking mechanism somewhere shared.
At scale it becomes more complex, but the overall principle still holds, though you may want to separate scheduling aka. the various distributed cron's.
As for load balancer, I agree it needs standardising, but it really is a small enough issue for smaller setups that it's fairly easy to solve. I've come to rely on setups that uses Nginx and rewrites incoming hostnames to a local hostname (e.g. foo.com => foo.com.service) and use resolvers in Nginx to point them to a SkyDNS instance. That means all the container config needs to do is set/delete Etcd keys to receive traffic for a given hostname.
But there's several load balancers out there that now can pull config directly from a suitable shared datastore, be it Redis or Etcd etc.
You might like to look at Kubo[1], which was jointly built by Googlers and Pivots to make this more manageable. We built it based on requests from joint customers who want BOSH as their infrastructure-level platform and the choice of Cloud Foundry or Kubernetes for application-level platform.
[1] https://github.com/pivotal-cf-experimental/kubo-deployment
edit: forgot my usual disclosure. I work on Cloud Foundry for Pivotal, though not on Kubo.
(Also, it saves syllables vs the alternatives)
I'm an HNer, I know of mature solutions that I trust, therefore Socrates is mortal.
If you want to keep things up to date, there are many examples scripts out there that either 1) checking to see if your base container is out of date (if someone updated jessie:8, you should rebuild) and 2) checking if any packages are out of date (sometimes limiting to just security updates) meaning you run apk, yum, apt, etc. within each container (or extract their package lists and examine them outside the container).
You gain a lot in the isolation, but debugging can become more difficult and you've introduced new areas of dependency rot.
I generally like containers, but hate how there are so many different orchestration formats (k8s, marathon, nomad, swarm, etc.) many different network layers (weave, flannel, etc.) and many different ways to package/fit components together (CoreOS, DC/OS, etc.)
The ecosystem is messy.
With such a system in place, pulling updates could be pinned to a particular API call, with a particular payment address, which itself was attached to a particular data store. In theory, as long as you had access to the wallet that did the build before, you'd be able to do the same build at a later date.
I'm greatly simplifying this here, but my conclusion after getting a crypto payment OpenStack instance launcher PoC working was that applying crypto payments to deployment methods basically provided a lightweight federated layer that allowed logically linking processes, similar to what you would get with a Zapier/Apigee/IFTTT knockoff. Applied to microservice architectures it would allow deployments to be both trustworthy and repeatable at later dates.