Is this "fixing"? The proper fix would be to use a seamless deployment by either spawning a new vm with new version, then redirecting the traffic once it's ready. Or by doing the same with a new version of the app server on the same host and replacing the socket when ready.
Otherwise, the requests will potentially wait a long time to be handled while the app starts up.
In this case it's mostly for experiments and stuff. And spawning a new vm etc is slow for that kind of thing. This takes less than a second (I just merge to master and it deploys).
also! I worked at a place where this was an issue for years. We eventually moved to Kubernetes, but it would have been nice to know this fix until then for the qa/dev environments where stuff was constantly deploying.
You have to make a request while the upstream is down. So do the restart, then make a request right away. If your service restarts quickly it'll be hard to reproduce.
To me the most appropriate way (when not dealing with kubernetes, but rather just vms) is to be able to mark specific server to not receive any new requests, wait for the requests to stop coming in and then deploy.
The article mentions nginx returning 502 when a backend is down during a deploy. This a problem in the cases where restarting the backend isn't atomic.
If the backend is able to perform a "reload" (vs. a slower stop/start or restart) and/or replace the old instances with the new when they're up -- assuming that's an atomic operation -- nginx will not return a 502.
If one is rapidly iterating and prefers having a somewhat cleaner stop/start, the advice can be helpful even though it masks/papers over a problem (the backend being down).
Detecting a failure and minimizing the request wreck that it causes is different from handling a deploy. The second one should cause no wrecks because your deploy management system should be able to tell your proxy layer to drain the target of deploy to allow the deploy to proceed when a target has no traffic
15 comments
[ 1418 ms ] story [ 3257 ms ] threadOtherwise, the requests will potentially wait a long time to be handled while the app starts up.
IIS web server has this but not nginx :(
If the backend is able to perform a "reload" (vs. a slower stop/start or restart) and/or replace the old instances with the new when they're up -- assuming that's an atomic operation -- nginx will not return a 502.
If one is rapidly iterating and prefers having a somewhat cleaner stop/start, the advice can be helpful even though it masks/papers over a problem (the backend being down).