49 comments

[ 4.8 ms ] story [ 81.9 ms ] thread
Error 403. Did he disable the last back end server in the pool?

See info below, correct link: https://www.claudiokuenzler.com/blog/1240/haproxy-how-to-dis...

I did (I am the owner of the blog). The blog article is on www.claudiokuenzler.com - not on this kakoku URL. I have no idea why Hacker News would change the link to another domain. Maybe someone knows this?
email dang (hn@ycombinator.com). He and his team will be able to fix this in no time.
Thanks for the hint! It may have been my error as well as I just discovered that copy cat URL before and blocked it. May have posted the copy-cat URL myself, who knows. Anyway I just posted a new submission with the correct link.
Changed now from https://gby.kakoku.online/blog/1240/haproxy-how-to-disable-e.... Thanks to both of you!

Our software does change links sometimes (it follows redirects and also uses the canonical URL when it finds one), but from the logs it doesn't look like any of that happened here.

thx, appreciate it! still need to figure out what this okaku domain does though...
(comment deleted)
hatop is a great ncurses interface to this from cli. https://github.com/feurix/hatop

Can also be done remotely https://github.com/Wirehive/haproxy-remote

Indeed! HATop is great to show the status, traffic, HTTP codes, errors or number of connections. It is also very simple to enable/disable HAProxy backends, for example:

  hatop -s /var/run/haproxy/example.sock
Use the arrows to select a backend server and press:

   F9 - Enable a backend server (Status: UP)
  F10 - Disable a backend server (Status: MAINTENANCE)
For Apple keyboards it is fn+F9 and fn+F10.
(comment deleted)
I prefer commenting out the entry referencing the server/node directly in the haproxy.cfg or even better, the code used to generate it und version control (Config-as-Code).
Yes but a reload can still cut a KEEP ALIVE established connection (observed in older HAProxy 1.6). Probably solved in more recent versions though.
This may be resolved in newer versions but definitely bit me before in older ones. I just have a script that drains off the backends we need before putting them in maintenance mode. Makes working on individual hosts much easier.
also, old haproxy process (with opened connections) still does checks which might be undesirable at scale.

as such we try to avoid reloading haproxy as much as possible. you can even renew a cert on-the-fly uploading the new cert via the admin socket.

Seems a little heavy weight for my "remove from LB, update code, add back to LB" Ansible script. :-)
An alternative I like a bit more than, `remove, update, add back` is to set a signal handler to close the listening socket, wait for current requests to finish, and then exit.

Assuming you initialize every service you depend on at program start and not in the critical path and that health checks only function after this setup work has been done it simplifies operations a bit and decreases the liklhihood of configuration errors (a drained server in a load balancer) from persisting.

That sounds clever and interesting, but in my case I'm probably dealing with ~20 services, some we build some third party, so the "remove from load balancer then update" trick works well.
The irony:

```

403 Forbidden

nginx

```

IMO not really - a valid architecture is nginx - haproxy - app servers
IME haproxy and nginx can overlap a lot as balancers/rewriter; having both of them adds no real world benefits, unless you have some special case where you need one special, unique feature from both of them.
> having both of them adds no real world benefits

This depends a great deal on the architecture of your application server and traffic load. There are a lot of reasons that it is nicer to use a queue in front of your load balancer, rather than use queues in your application server, or worse, use the operating system queue.

You must have SSL termination, logging, load balancing, and queuing. You probably want to modify headers and interpret cookies as well.

HAProxy is a queue that feeds into a load balancer. Nginx is a buffer that feeds into a load balancer (IIRC).

One useful property that HAProxy can ensure is no more than 1 request going to a server at a time. I believe (but potentially incorrectly) that nginx will fail requests, rather than "queue" them, if it tries to ensure this property. HAProxy also historically had health checks for backend servers, while for nginx it was a paid premium feature.

I could be wrong about nginx not being able to queue. It has been a while.

For my mental model, nginx is the right choice for SSL termination, logging, request mangling, interpretation of cookies and loadbalancing based on request information (for example choosing haproxy instances based on a domain name). HAProxy is the right tool for queuing and load balancing to servers that will actually fulfill the request.

Disclosure: Community contributor to HAProxy, I help maintain HAProxy's issue tracker.

> For my mental model, nginx is the right choice for SSL termination, logging, request mangling, interpretation of cookies and loadbalancing based on request information (for example choosing haproxy instances based on a domain name).

HAProxy can do all that and IMO it also does it better.

Personally I chain HAProxy and nginx in reverse order: HAProxy exposed to the Internet, doing all the heavy lifting. (Multiple) nginx with minimal config as a static file server and FastCGI gateway behind HAProxy.

see also: https://news.ycombinator.com/item?id=27253579

I've done a similar architecture as well a couple of times. Both setups use multiple haproxy instances (auto-scaling group), multiple backend app servers, and each haproxy server has a local nginx instance.

Nginx is used for serving some static error pages that live outside of the backend applications haproxy talks to (eg: "Customer domain not recognized"). There's some hacks to kind of make this work in haproxy but it's much simpler with an actual HTTP content server.

Nginx also works well as a caching proxy, and what's really cool with haproxy is it's easy to have most requests go directly to the application server (avoiding an unnecessary proxy hop), but just pattern match certain types -- eg regex match `.jpg$` -- to go through the nginx cache.

I've also used a similar technique to have per-domain custom images overriding specific URL paths (eg `/logo.png`), which was a really quick way to allow some customization of a multi-tenant application without having to make major code modifications to support it (only used for a tiny handful of customers).

Having both makes perfect sense if you like haproxy for what it's good at but also want a web server.
I said using both of them as a balancers/rewriter (aka proxy) in the same architecture. nginx as a webserver behind haproxy is perfectly fine obviously.
There’s other ways than using telnet??? I thought telnet was the recommended way.
Haproxy is definitely under-appreciated for the amount of heavy lifting it does in a high-scale deployment.
A lot of just don't realize how and where it's used. I'm almost 100% positive that Microsoft uses HAProxy (open source or commercial license I couldn't tell you) to power App Services on Azure. Yet you'll get friction from CTOs at "Microsoft shops" for choosing HAProxy to use yourself.
> I'm almost 100% positive that Microsoft uses HAProxy (open source or commercial license I couldn't tell you) to power App Services on Azure.

I think you'll find they're using ARR and not HAProxy for the "batteries included" load balancer. You can tell this from the cookie names "ARRAffinity" and "ARRAffinitySameSite".

Almost certain that AWS uses HAProxy behind their ELB (Elastic Load Balancer) service. But if you know HAProxy, you feel very limited when using ELB.
We built a fantastic CDN on top of HAProxy, and all these nifty tricks are only scratching the surface of its potential.

The ease with which you can make real-time reactions to threats and conditions is really second to none, and with the ability to truly "hitless" reload (reload without affecting in-flight requests and connections) it makes those have immediate impact and no customer impact.

Sometimes the capabilities make the config a bit inscrutable, but other than that, I can't recommend it enough.

Newer versions also have few more tricks up the sleeve, like you can probe and act upon client's TCP RTT and RTT variance (because apparently newer kernels just track that info and it can be accessed from userspace).

We ended up using HAPRoxy as front for pretty much everything (even if techncially slapping just Nginx would be simpler at first), just because it often happens that this and that needs to be added (stuff like adding proper headers when devs don't want to figure out how to make their '00s CMS emi the '20s security headers).

It can also talk directly via unix socket to the backends so you don't even need to have a bunch of ports open if you want to front something.

Have you used the ngx_lua module (or OpenResty) with Nginx at all? I'm curious if there was something HAProxy does that can't be done with OpenResty, because that is crazy full-featured. Basically the main reason I haven't played around with HAProxy is just because I haven't ever felt like I was missing anything with OpenResty/Nginx
HAProxy also has Lua support.

The general answer here, is no, I don't think you can do much in HAProxy that you can't do with OpenResty, but with HAProxy you generally don't have to pay the overhead of Lua (and HAProxy tends to be very well optimized, so the difference can be significant at high throughput of requests / bytes).

One notable thing is that OpenResty, unlike HAProxy, uses LuaJIT, which is dramatically faster.
HAproxy is awesome. We have used the socket for ingress rate limiter for reasonably successful VoD service.
Wow. Nice. I'm working on a similar project atm. Do you have any info on how to configure the socket?
you can also add

    stats admin
to the stats backend to enable ability to do that from statistics UI address. Just make sure it is at the very least behind a password
HAProxy has blown my mind. I'm using it to build a deployment tool for Joystick [1] and it's a real treat to work with. I utilize this specific feature (enabling/disabling servers on the fly) and it worked on the first swing without issue. Made scaling a cluster of servers fairly effortless.

[1] https://github.com/cheatcode/joystick