The difference is that HAProxy has a monitoring page to see what is configured and where connections are going. Nginx is a black box unless you pay them ~ $2000 for the pro edition.
Given two load balancers, one can load balance 20K requests per second but has good monitoring, the other can load balance 500K requests per second but has bad monitoring. Which is the better load balancer?
Note that I'm not implying these figures are true of either HAProxy or nginx, just pointing out that the OP is probably correct that the number one feature of a load balancer is its ability to load balance. Good monitoring makes it easier to use, though.
All is good when it's working well, but when something's not right... well, you really wish that you had that monitoring.
The truth is that most load balancing solutions are capable of handling _a lot_ of traffic, way way more than the majority of the people need. So performance is not a huge concern, typically.
However, if you are at the level where it does actually matter and want to use nginx, chances are, you'll have no problem paying for the premium version.
They both have the same performance for HTTP(S) traffic, both software are the same in their algorithms and performance critical aspects. The limit is set by the harware, they will either saturate the NIC or the CPU.
HAProxy has a status page to see how many requests are processed and going where. nginx doesn't.
> the other can carry 1000 but you are flying blind.
This is a bad analogy, because 'flying blind' in a plane means you can't use it effectively. Being able to see out of a plane isn't a 'bonus'; it can't be used without this feature. Whereas plenty of people can and do use a load-balancer without heavy monitoring. (not to mention that there are strong moves towards self-driving transportation at the moment)
The analogy also doesn't work because a loadbalancer that can handle 1000 units is going to have far fewer problems that are in need of diagnosis than one whose limit is 100 units; when the traffic level is between 100 and 1000. When the traffic is at 100, then yes, you're going to want monitoring on the 100-unit LB, because you're trying to squeeze extra performance out of it. Whereas the 1000-unit one isn't even sweating at that point.
I was expecting this post to be some kind of benchmark/comparison between the two. A better title would be: "Don't use nginx unless you're paying for it".
While the OP is about metrics when it comes to load balancing IPVS from the LVS project http://www.linuxvirtualserver.org/software/ipvs.html is often overlooked. If you're trying to balance a lot of connections/traffic (and aren't/don't want to buy a hardware load balancer) you'll get much better performance with IPVS than HAProxy or nginx.
I've not looked at LVS for a while, does it do SSL termination too? (Not sure as it's Layer4). We generally use haproxy for this reason in addition to using it for balancing.
I've used IPVS a lot in the past and it worked well in a lot of ways. We had a number of problems with ldirectord getting wedged after months of uptime, and the tools are all a little primitive, but IPVS itself works quite well. The amount of traffic it can handle, especially in the "direct routing" configuration, is stunning.
But, I can't imagine going back to it from haproxy. Haproxy allows so many other things including nice status page, agent and service checks, SSL termination, great monitoring (we use Influxdb/telegraf/grafana and Icinga2), and really advanced routing of requests (SNI, path, pretty much anything in the requests).
I guess it depends what you're load balancing, you do get pool checks from ipvs and node_exporter has an ipvs collector if you're using prometheus (we are). As you say the amount of traffic it can handle in comparison in stunning and especially if you're load balancing over a large number of ports it's less to configure than HAProxy, but it works at level 4 so if you need HTTP stuff it's not much help.
Whenever I see an article that says “you should NEVER do x” I typically treat it with a large pinch of salt.
Plenty of people, myself included, use nginx as a load balancer and it’s absolutely fine. I’m sure there are lots of use cases where HAProxy would be a better choice, but there are also many use cases where nginx is at least as good a choice.
While I get your point and don't really agree with spoiling books here to make one, firstly. Secondly, I don't really think it applies: this isn't literature; the title relevance to the content is more important than being creative.
Jay brushed the remnants of the Nginx sticker from his laptop. "I will not be enthusiastic for fake open source software" he said. " I must thank Hacker News commentators for saving me the time..."
I'm getting $2500 for a single license at the lowest level of support. They don't specify USD, so possibly I'm getting AUD because of my location, but usually a global vendor will specify if they're giving prices in non-USD.
Per server, no less. Enterprise pricing, but sucks for SMBs.
I'm pretty much in love with Nginx' simplicity and capabilities as a swiss army knife for all kinds of HTTP magic.
I had some doubts about Nginx' direction and feature development, but most really great features (like stream proxy with SNI support) make their way into the open source release. Built in monitoring sucks though. There are options for better monitoring your requests with Prometheus.
This project implements prometheus metrics using Lua scripting inside Nginx:
This project is indended as a sidecar service to nginx and receives access logs and timings using syslog protocol (Nginx has a native syslog over UDP capabilities) so no scripting inside Nginx is necessary:
I honestly don't know, I never made serious performance tests. The external exporter can make use of every variable that nginx is able to use in a log_format. I guess the internal Lua scripting has access to even more data.
Performance wise I assume the external exporter has less impact on Nginx' operation because it doesn't keep shared state or run code inside Nginx at all (see https://github.com/knyar/nginx-lua-prometheus#caveats), all Ngnix does is push out log records over syslog/UDP. If the exporter should overload that doesn't concern Nginx at all, what happens is that observations get dropped.
haproxy tends to be lighter weight and higher performance. Also, haproxy gives (as this article mentions) a lot more information regarding status information, which can be used to monitor and manage the applications being load balanced.
Willy also was the long term maintainer of the 2.4 Linux kernel (yes... 2.4), and does a lot of maintenance with the long term 2.6 release. So many embedded companies rely on his trees for their devices it isn't even funny.
Is stream proxy with SNI supported by open source version of Nginx? I can't find any documentation on how to route TCP traffic based on SNI but I know haproxy can use SNI and ALPN for that.
so glad to find this - will look at incorporating with our prometheus setup. We are using OpenResty so the luajit stuff is already in place. I am excited to see what this can do for us. thanks
It looks like you have some really good experience in tooling around metric collection surrounding NGINX.
Edit: For reference, here are all the metrics we can collect...and I believe it to be a fairly complete list of notable metrics pertaining to NGINX performance:
At the moment we do not have an on-prem or self-hosted solution.
Currently, we are working on the NGINX Controller which will have an embedded Amplify within it to drive it's telemetry needs (and will be available in addition):
If you are interested in an on-prem version I would love if you could give it a try. If it's something you think you would find useful in an on-prem scenario then please leave us feedback in Intercom!
The more people that ask for it the more likely our product owners will prioritize it :)
what's sad about haproxy that it still does not support H2 (HTTP/2). I mean a load balancer should at least support it on the front end.
Well haproxy is still great and one of the best load balancers out there.
But it's still sad that it misses h2 on the frontend.
(Well backend would be cool, but there aren't that many load balancers that can do that anyway).
I'm pretty sure haproxy 1.8 will at least give experimental support! which would be awesome.
TLDR; this whole article is a joke. The guy is not capable of extracting stats out of his nginx instance, so "Conclusion: Avoid nginx at all costs". End of article.
I must admit I'm not that huge of a fan of nginx myself, either - The "easy to read, easy to reason about, declarative configuration"-premise breaks down pretty hard when your application setup gets moderately complex. There's just too much obscure magic woven into innocent-looking configuration directives, and as a result of this, more than half of the world gets things wrong (by (ab)using "if" and the like). On top of that, as the article states, introspection into what is going on is laughable in the FOSS release.
Still, we use nginx to terminate https at work, and it's doing a very fine job at that - much better than pound, which we used before. If it weren't for Varnish directly behind it, however, it would very often be supremely difficult to debug what's actually going on between our HTTP backends and UAs if the need arises.
I hit this problem too - we had >100k lines of nginx config. My answer was to write a tool to simulate what happened to a request, logging the decisions made each step. It was limited in what config I'd programmed it to understand but did most of what we needed (rewrite rules, access control, proxy_pass ...). I also intended it to be used for regression testing - you could set up a yaml file with requests and specify eg, return this proxied response if the proxied request headers match X.
Other efforts I'd seen at automated testing of nginx config relied on testing in a VM, but of necessity used different listen addresses etc. They looked complex to set up and weren't even testing the config as shipped...that's what I was aiming for.
Since then we've moved away from that config, and I'm in a different role...keep meaning to see if we can open source that code. However, nginx being such a moving target, with lua config and such, I was never sure if it would find an audience.
Docker with docker compose is good for this. Nginx can listen on ports 80 and 443, other containers on the same network can reach it on those ports. I have a setup which takes env vars, so it's not quite the same as shipped, but I can test the whole process of getting a cert from LetsEncrypt, for example.
(late reply but...) we did actually look at that; someone had a blog post about using docker for testing nginx when we did this (~3yr ago). You hit a number of problems. Our nginx listened on multiple addresses and ports, which weren't equivalent - external https/http arrived on one address, internal traffic from across our clusters arrived on another, with different handling. So, we have to have docker listen on multiple addresses too, or in multiple instances. Then we have the outgoing requests, proxied to various haproxy, varnish instances, themselves on various ports. You can get round this by messing with dnsmasq and routing on your docker host...or avoid that by reorganizing all of the outgoing config so that they talk to a stub server with canned test responses. But, we had thousands of lines of handwritten nginx config as well as tens of thousands of lines of generated config...this was going to be a major effort.
And even with all that effort, it didn't give us enough information. You'd try a request on some config and scratch your head to figure out why it didn't end up being processed in the right place. Run it in the simulator though, and you'd find the exact line that had rewritten your request so it didn't reach the line of config you thought was going to kick in...
Probably the best approach would have been to patch nginx itself to add the logging we wanted, and have it handle all of those external addresses differently. The challenge there would have been getting the same depth of traceability with nginx doing things like converting rewrite rules to bytecode. But, you could probably get 80% of the way there and avoid bugs from our system not accurately re-implementing their algorithms.
We moved away from nginx to envoy https://envoyproxy.github.io/ .
With envoy you get best of both the world with hot reload.
Envoy excels in metrics as well as remote configuration.
Recently discovered Traefik (https://traefik.io/) which is a highly performant reverse proxy written in Go. Supports just about every popular backend in existence (Docker, Kuber, Eureka, Consul and whatnot), supports metrics through statsd, Prometheus, Datadog or RYO via REST. Has a pretty good Admin UI but also comes with a REST API to integrate it in your own.
Also it is entirely free and open source. Yes you are reading it correctly: Within your edge environment you are able to get pretty much all the features of a $2500+ per instance NGINGX Plus deployment with equal performance for nothing but a little investment in time (unless you need/want commercial support, ofcourse).
nginx, while being a black box, can at least do logging. The haproxy logging in docker can't log to stdout since over three years without a workaround involving a linked rsyslogd (https://github.com/dockerfile/haproxy/issues/3).
On the other hand, I like configuring a haproxy way, way more than configuring a nginx server. It's easier, and more concise to read.
And the official image does not provide for rsyslogd. I want an official image to at least support logging out of the box without having to resort to docker-compose or even weirder setups.
nginx has issues with formatting numbers/string and undefined values, which makes it harder to extract meaningful data from log. haproxy is easier to deal with.
Docker abstract away the logging facilities and the filesystem. It's a general issue with Docker, not the software. See the 4th answer in your issue for a solution.
> Docker abstract away the logging facilities and the filesystem.
No it does not, it expects stdout and stderr to be used and stores its output in a logfile to be replayed with "docker logs" command.
First solution in issue: mount /dev/log. Does not work when your host system is OS X or Windows, as you don't have access to the VM that Docker internally uses on non-linuxes. Also, on Linux it pollutes the syslog instead of docker log.
Second (the one that refers to a gmane post from 2010) is essentially the same as #1.
Third, rsyslog on host (or in a docker image): does not use docker log facility, and docker link is weird.
Fourth, use the alpine image which has a syslogd: works, maybe, but is subject to changes in the Dockerfile (e.g. when the inner path of haproxy changes). Also I don't touch Alpine images because I can't use the standard Debian/Ubuntu tooling in case there are network/firewall issues that can best be debugged from inside the container by docker exec -it haproxy bash.
> Docker abstracts away the network and the filesystem.
Not neccessarily the network, if you're using --net=host.
Docker does not screw with writing logs to files, to stdout or via UDP networking. When it comes to unix sockets or syslog, depends if the target is inside the container (which works fine AFAIK) or outside (e.g. due to bind-mounting /dev/log or such), but I have not tried the latter.
> I have a perception of nginx being slimmer than HAproxy and more focused on http/https than HAproxy
Oddly enough, I have the exact opposite perception. HAProxy doesn't even support logging to files because that would "block" the event loop, so it delegates to syslog. That seems fairly slim to me. The entire project is written with no compromises to support extreme numbers of connections efficiently. Nginx can do this as well, but has to be tuned a lot more to get to the same place, and I'm convinced if both were properly tuned and put in the same environment, HAProxy would come out ahead.
That said, if you're at a place where you already need/use Nginx, it might not make sense to use HAProxy if you can re-use your existing Nginx instances. However if you're in the market for a load balancer and Nginx isn't being used already, definitely check out HAProxy. It's pretty simple to configure.
EDIT: Another point for HAProxy: Recently, the job system we run (beanstalkd) was slowing to a crawl. I needed to find out where the slowdown was. The easiest way was to route the worker connections through HAProxy and enable logging and the HTTP stats page. I was able to determine the problem was in beanstalkd itself, not our workers (or a network issue). Sure there are a number of debugging proxies out there, but HAProxy was already installed and the logs it emits are the perfect balance of information so you can pick things apart without being overwhelmed. It really helped. Nginx wouldn't have been as useful, and would have taken a much longer time to configure.
> Oddly enough, I have the exact opposite perception.
Ditto, especially if you consider the OpenResty "variant" of Nginx.
Also, I realize Nginx does a good job of being a load balancer / reverse proxy, but to me it will always be a "web server" first. HAProxy on the other hand is built and optimized specifically to be a "load balancer / reverse proxy" first.
haproxy is able to preserve source ip by injecting proxy protocol. nginx is capable of reading proxy protocol but not injecting it.
which is why haproxy is perfect as the ingress for your web application. Especially if you are using something like kubernetes that will use an overlay network.
unfortunately, the momentum in kubernetes is to leverage nginx as an ingress.
> the momentum in kubernetes is to leverage nginx as an ingress.
There are many different groups, Nginx in fact has their own ingress controller [1] separate from K8S nginx ingress controller. HAProxy is older and harder to deal with for L7 ingress routing as compared to Nginx.
The Istio project (using Envoy) [2] is gaining speed quickly though to become a much better ingress and internal proxy.
I only skimmed the article but I agree with the conclusion. I evaluated both HAProxy and Nginx for a high-volume/low-latency load balancer cluster (500k requests/s) and HAProxy simply beat the pants off of Nginx. It has an extensive and well documented set of configuration options, tooling, and reporting, and it performed flawlessly on production (after much toiling). I couldn't ever quite get Nginx to handle the same load without falling apart at the seams.
I'm glad this has 2016 in the title! It's now 2017, and we have better options.
Nginx has been the best routing and traffic serving system for a while now, but it's showing its age. In addition to hiding metrics, HTTP 2 support is difficult, there's no clear path to grpc, and (as mentioned in tfa) there is no way to get stats out of it.
Envoy (envoyproxy.github.io) seems like the next thing. Nginx won because of the predominant patterns at the time, and it's still totally fine for serving lots of HTTP traffic. The problem is that if you're thinking microservices or rely on metrics to do your job or have elastic infrastructure, Envoy's feature set is the only real way to handle traffic routing in that world.
Disclaimer: I work for turbinelabs.io, and we're moving from nginx to Envoy to take advantage of a lot of these features to power our product. We've put a lot of effort into that decision, and believe it's the right one.
I haven't used Envoy yet, but i find it attractive that its made by a company that doesn't make money by selling enterprise features like Nginx+ does. Lyft doesn't care about that, so they have "Advanced Load Balancing" features baked in for free.
I thin nginx+ is handled really well, a lot of new features find their way into the open source version and for most people that version is all they ever need.
You don't call Envoy to make changes. You either update the config file, or you implement an API server with a specific set of endpoints, then have it serve the updated values back. It's a bit tricky, but entirely doable. https://envoyproxy.github.io/envoy/intro/arch_overview/dynam...
(You can do this in nginx, too, and it's similarly annoying. This is a big part of what we do at Turbine Labs -- a UI with a slider between blue and green, and a place to host all those stats, so you don't have to manage it!)
I'd use Houston in a heartbeat over nginx (currently flying blind, but need it for HTTP/2) but it's hard to compete with free. It's hard to read what your licensing/support model is like without signing up. Considered a Red Hat like support model?
turbinelabs.io is the whole of everything we're building on top of nginx/envoy. Our primary goal is to provide more powerful ways to release software like incremental blue/green deploys or canaries or production previews. That requires a solid data plane (nginx/envoy) and management plane (missing in nginx/envoy xDS APIs) underneath it to work. Nobody runs nginx because it's fun -- one of the logical use cases for a better nginx is better releases, which is what we work on.
The transition has been good! There's certainly a lot more complexity to implement with the Envoy APIs, but that's because there's a lot of power. Working with Matt Klein and the rest of the community has been really positive. We're excited about all the other patterns that unlocks, like circuit breaking and tracing. Those sorts of features are on the wrong side of the cost/benefit equation for most companies, but if you get them for free because you deployed Envoy as ingress and service mesh, they're great to have.
There are 4 Discovery Service APIs that you can implement: service (SDS), cluster (CDS), route (RDS), and listener (LDS). Colloquially called xDS for all of them.
Could you elaborate more on the HTTP2 support being difficult? Enabling it is a one liner in the relevant config file, so I'm guessing there must be some more subtle gotchas.
115 comments
[ 2.9 ms ] story [ 186 ms ] threadThe difference is that HAProxy has a monitoring page to see what is configured and where connections are going. Nginx is a black box unless you pay them ~ $2000 for the pro edition.
Monitoring is feature #1 of a load balancer.
A scooter can transport cargo. A truck can transport cargo effectively.
Note that I'm not implying these figures are true of either HAProxy or nginx, just pointing out that the OP is probably correct that the number one feature of a load balancer is its ability to load balance. Good monitoring makes it easier to use, though.
The truth is that most load balancing solutions are capable of handling _a lot_ of traffic, way way more than the majority of the people need. So performance is not a huge concern, typically.
However, if you are at the level where it does actually matter and want to use nginx, chances are, you'll have no problem paying for the premium version.
HAProxy has a status page to see how many requests are processed and going where. nginx doesn't.
Monitoring is essential to any infrastructure, regardless of its scale.
This is a bad analogy, because 'flying blind' in a plane means you can't use it effectively. Being able to see out of a plane isn't a 'bonus'; it can't be used without this feature. Whereas plenty of people can and do use a load-balancer without heavy monitoring. (not to mention that there are strong moves towards self-driving transportation at the moment)
The analogy also doesn't work because a loadbalancer that can handle 1000 units is going to have far fewer problems that are in need of diagnosis than one whose limit is 100 units; when the traffic level is between 100 and 1000. When the traffic is at 100, then yes, you're going to want monitoring on the 100-unit LB, because you're trying to squeeze extra performance out of it. Whereas the 1000-unit one isn't even sweating at that point.
we loadbalance nginx clusters with haproxy and we use filebeat & metricbeat to feed logs to an ELK stack.
That monitoring page is useless to us now and your logging & monitoring should be centralized to one place anyway.
For those who are on AWS, Network Load Balancer is also competitive with LVS.
ipvsadm -L will get you metrics in a human-readable format, /proc/net/ip_vs* has stats for monitoring systems to parse.
But, I can't imagine going back to it from haproxy. Haproxy allows so many other things including nice status page, agent and service checks, SSL termination, great monitoring (we use Influxdb/telegraf/grafana and Icinga2), and really advanced routing of requests (SNI, path, pretty much anything in the requests).
It allows for SSL protected custom domains for a user of a SaaS app in a really easy to do way.
We use it at Shopblocks extensively, and I've written a (brief) outline of roughly how we do it.
https://zando.io/post/haproxy-saas-custom-domains/
Plenty of people, myself included, use nginx as a load balancer and it’s absolutely fine. I’m sure there are lots of use cases where HAProxy would be a better choice, but there are also many use cases where nginx is at least as good a choice.
Conclusions don’t have to be in the title
I'm getting $2500 for a single license at the lowest level of support. They don't specify USD, so possibly I'm getting AUD because of my location, but usually a global vendor will specify if they're giving prices in non-USD.
Per server, no less. Enterprise pricing, but sucks for SMBs.
I had some doubts about Nginx' direction and feature development, but most really great features (like stream proxy with SNI support) make their way into the open source release. Built in monitoring sucks though. There are options for better monitoring your requests with Prometheus.
This project implements prometheus metrics using Lua scripting inside Nginx:
https://github.com/knyar/nginx-lua-prometheus
This project is indended as a sidecar service to nginx and receives access logs and timings using syslog protocol (Nginx has a native syslog over UDP capabilities) so no scripting inside Nginx is necessary:
https://github.com/markuslindenberg/nginx_request_exporter
(I'm the author of this exporter)
Is there any difference in what data is able to be exported?
Performance wise I assume the external exporter has less impact on Nginx' operation because it doesn't keep shared state or run code inside Nginx at all (see https://github.com/knyar/nginx-lua-prometheus#caveats), all Ngnix does is push out log records over syslog/UDP. If the exporter should overload that doesn't concern Nginx at all, what happens is that observations get dropped.
Also note that Willy Tarreau has been a Linux kernel guy for eons and does a lot of upstream kernel patches, some of which haproxy benefit from. Most recent example: https://github.com/torvalds/linux/commit/3979ad7e82dfe3fb94a...
Willy also was the long term maintainer of the 2.4 Linux kernel (yes... 2.4), and does a lot of maintenance with the long term 2.6 release. So many embedded companies rely on his trees for their devices it isn't even funny.
Disclaimer: I am on the Amplify team at NGINX.
https://www.nginx.com/products/nginx-amplify/
It looks like you have some really good experience in tooling around metric collection surrounding NGINX.
Edit: For reference, here are all the metrics we can collect...and I believe it to be a fairly complete list of notable metrics pertaining to NGINX performance:
https://amplify.nginx.com/docs/guide-metrics-and-metadata.ht...
Is there a self hosted version?
Currently, we are working on the NGINX Controller which will have an embedded Amplify within it to drive it's telemetry needs (and will be available in addition):
https://www.nginx.com/products/nginx-controller/
If you are interested in an on-prem version I would love if you could give it a try. If it's something you think you would find useful in an on-prem scenario then please leave us feedback in Intercom!
The more people that ask for it the more likely our product owners will prioritize it :)
Well haproxy is still great and one of the best load balancers out there. But it's still sad that it misses h2 on the frontend. (Well backend would be cool, but there aren't that many load balancers that can do that anyway). I'm pretty sure haproxy 1.8 will at least give experimental support! which would be awesome.
" - TLS NPN and ALPN extensions make it possible to reliably offload SPDY/HTTP2 connections and pass them in clear text to backend servers"
Still, we use nginx to terminate https at work, and it's doing a very fine job at that - much better than pound, which we used before. If it weren't for Varnish directly behind it, however, it would very often be supremely difficult to debug what's actually going on between our HTTP backends and UAs if the need arises.
Other efforts I'd seen at automated testing of nginx config relied on testing in a VM, but of necessity used different listen addresses etc. They looked complex to set up and weren't even testing the config as shipped...that's what I was aiming for.
Since then we've moved away from that config, and I'm in a different role...keep meaning to see if we can open source that code. However, nginx being such a moving target, with lua config and such, I was never sure if it would find an audience.
And even with all that effort, it didn't give us enough information. You'd try a request on some config and scratch your head to figure out why it didn't end up being processed in the right place. Run it in the simulator though, and you'd find the exact line that had rewritten your request so it didn't reach the line of config you thought was going to kick in...
Probably the best approach would have been to patch nginx itself to add the logging we wanted, and have it handle all of those external addresses differently. The challenge there would have been getting the same depth of traceability with nginx doing things like converting rewrite rules to bytecode. But, you could probably get 80% of the way there and avoid bugs from our system not accurately re-implementing their algorithms.
Also it is entirely free and open source. Yes you are reading it correctly: Within your edge environment you are able to get pretty much all the features of a $2500+ per instance NGINGX Plus deployment with equal performance for nothing but a little investment in time (unless you need/want commercial support, ofcourse).
Cap Gemini UK apparently is using it within projects [1], GitLab is using it within their infrastructure [2] and Reevoo is using it aswell [3].
Katacoda [4] and Play-with-Docker [5] both have sandbox environments available which allow you to play with it.
[1] https://hackernoon.com/kubernetes-ingress-controllers-and-tr...
[2] https://about.gitlab.com/2017/07/11/dockerizing-review-apps/
[3] https://www.youtube.com/watch?v=aFtpIShV60I
[4] https://www.katacoda.com/courses/traefik/deploy-load-balance...
[5] http://training.play-with-docker.com/traefik-load-balancing
On the other hand, I like configuring a haproxy way, way more than configuring a nginx server. It's easier, and more concise to read.
And the official image does not provide for rsyslogd. I want an official image to at least support logging out of the box without having to resort to docker-compose or even weirder setups.
nginx has issues with formatting numbers/string and undefined values, which makes it harder to extract meaningful data from log. haproxy is easier to deal with.
Docker abstract away the logging facilities and the filesystem. It's a general issue with Docker, not the software. See the 4th answer in your issue for a solution.
No it does not, it expects stdout and stderr to be used and stores its output in a logfile to be replayed with "docker logs" command.
First solution in issue: mount /dev/log. Does not work when your host system is OS X or Windows, as you don't have access to the VM that Docker internally uses on non-linuxes. Also, on Linux it pollutes the syslog instead of docker log.
Second (the one that refers to a gmane post from 2010) is essentially the same as #1.
Third, rsyslog on host (or in a docker image): does not use docker log facility, and docker link is weird.
Fourth, use the alpine image which has a syslogd: works, maybe, but is subject to changes in the Dockerfile (e.g. when the inner path of haproxy changes). Also I don't touch Alpine images because I can't use the standard Debian/Ubuntu tooling in case there are network/firewall issues that can best be debugged from inside the container by docker exec -it haproxy bash.
There are many way to write logs, to files, to syslog, to unix socket, to stdout, to UDP sockets. Docker screws with most of them.
Not neccessarily the network, if you're using --net=host.
Docker does not screw with writing logs to files, to stdout or via UDP networking. When it comes to unix sockets or syslog, depends if the target is inside the container (which works fine AFAIK) or outside (e.g. due to bind-mounting /dev/log or such), but I have not tried the latter.
The vts module has been updated in recent months to support more granular reporting of request times, but it's had everything we need for a while.
It exposes metrics either as an html page or a json api.
So with vts I can finally replace last instance of haproxy which serves as a core LB between all services (because it has stats).
If I'm load balancing http or terminating https I'm not going to use HAproxy. HAproxy is very powerful but overkill.
* I have a perception of nginx being slimmer than HAproxy and more focused on http/https than HAproxy
* Related to the aforementioned point; I have a perception of HAproxy being able to handle many OSI layers compared to nginx which only handles one.
Oddly enough, I have the exact opposite perception. HAProxy doesn't even support logging to files because that would "block" the event loop, so it delegates to syslog. That seems fairly slim to me. The entire project is written with no compromises to support extreme numbers of connections efficiently. Nginx can do this as well, but has to be tuned a lot more to get to the same place, and I'm convinced if both were properly tuned and put in the same environment, HAProxy would come out ahead.
That said, if you're at a place where you already need/use Nginx, it might not make sense to use HAProxy if you can re-use your existing Nginx instances. However if you're in the market for a load balancer and Nginx isn't being used already, definitely check out HAProxy. It's pretty simple to configure.
EDIT: Another point for HAProxy: Recently, the job system we run (beanstalkd) was slowing to a crawl. I needed to find out where the slowdown was. The easiest way was to route the worker connections through HAProxy and enable logging and the HTTP stats page. I was able to determine the problem was in beanstalkd itself, not our workers (or a network issue). Sure there are a number of debugging proxies out there, but HAProxy was already installed and the logs it emits are the perfect balance of information so you can pick things apart without being overwhelmed. It really helped. Nginx wouldn't have been as useful, and would have taken a much longer time to configure.
Ditto, especially if you consider the OpenResty "variant" of Nginx.
Also, I realize Nginx does a good job of being a load balancer / reverse proxy, but to me it will always be a "web server" first. HAProxy on the other hand is built and optimized specifically to be a "load balancer / reverse proxy" first.
which is why haproxy is perfect as the ingress for your web application. Especially if you are using something like kubernetes that will use an overlay network.
unfortunately, the momentum in kubernetes is to leverage nginx as an ingress.
Moreso than the GCP loadbalancing or the relatively new AWS transit loadbalancers?
However, in the limited work happening on lb increases, it's nginx rather than haproxy.
There are many different groups, Nginx in fact has their own ingress controller [1] separate from K8S nginx ingress controller. HAProxy is older and harder to deal with for L7 ingress routing as compared to Nginx.
The Istio project (using Envoy) [2] is gaining speed quickly though to become a much better ingress and internal proxy.
1. https://github.com/nginxinc/kubernetes-ingress
2. https://istio.io
Weird.
Traefik: reverse proxy built in Go with dynamic backends and modern integrations, good replacement for most situations. https://traefik.io/
Envoy: fast C++ L3/L7 proxy with some great features, http/2 support both ways, websockets, grpc, advanced health checking and load balancing, lots of metrics and integrations, my default recommendation now. https://envoyproxy.github.io/ + good comparison page: https://envoyproxy.github.io/envoy/intro/comparison.html
You supposed pick the right tools for a project not a project for the tools.
https://github.com/envoyproxy/envoy/issues/128
Nginx has been the best routing and traffic serving system for a while now, but it's showing its age. In addition to hiding metrics, HTTP 2 support is difficult, there's no clear path to grpc, and (as mentioned in tfa) there is no way to get stats out of it.
Envoy (envoyproxy.github.io) seems like the next thing. Nginx won because of the predominant patterns at the time, and it's still totally fine for serving lots of HTTP traffic. The problem is that if you're thinking microservices or rely on metrics to do your job or have elastic infrastructure, Envoy's feature set is the only real way to handle traffic routing in that world.
Disclaimer: I work for turbinelabs.io, and we're moving from nginx to Envoy to take advantage of a lot of these features to power our product. We've put a lot of effort into that decision, and believe it's the right one.
(You can do this in nginx, too, and it's similarly annoying. This is a big part of what we do at Turbine Labs -- a UI with a slider between blue and green, and a place to host all those stats, so you don't have to manage it!)
Put the green servers in one cluster, the blue servers in another cluster. Then use the route configuration to switch between the two.
https://envoyproxy.github.io/envoy/configuration/http_conn_m...
The transition has been good! There's certainly a lot more complexity to implement with the Envoy APIs, but that's because there's a lot of power. Working with Matt Klein and the rest of the community has been really positive. We're excited about all the other patterns that unlocks, like circuit breaking and tracing. Those sorts of features are on the wrong side of the cost/benefit equation for most companies, but if you get them for free because you deployed Envoy as ingress and service mesh, they're great to have.
https://envoyproxy.github.io/envoy/intro/arch_overview/dynam...