When I'm looking for a new job I quite like to ask "Do you deploy on Fridays?" when it comes to the "any questions?" part of the interview process.
Obviously it doesn't really work for companies that do continuous deployment, but for those that don't it provides great insight. Often you'll here an interviewer praise the fact the company doesn't do Friday deploys "because if it goes wrong that could eat in to our developer's weekends, and we like our developers so we'd never do that". That's the wrong answer. The right answer is "Of course, we trust our deploy process."
You can trust your deploy process and still deploy bad code because sometimes something gets missed in design, review, or testing. Why buy that headache for what is probably a marginal gain? (no matter how much we want to think we're changing the world.... that deploy is probably marginal gain that can wait until Monday)
The question you should be asking is: what's so important that we need to rush on a Friday? Can it wait to Monday?
Now, I agree with the premise of "trust your process" but if you've been into this field for a while, you know things will break.
Once a QA guy I worked with, came back from his weekend with a new test case/scenario that resulted in a critical issue right before we were ready to deploy. Not even CI/CD can help you with that.
This post seems written assuming that engineers are calling all the shots. If you work at a company like that, great, I envy you.
However in a company where engineering's desire for a stable deploy pipeline with excellent visibility is matched up against business' insistence that all dev hours go into features and SEO improvements and every sprint issue regarding non-user-facing-improvements has to be fought for tooth and nail, "don't deploy on Friday" is a very reasonable compromise for avoiding weekend fires.
I think this argument assumes that frequent deploys and good CI/CD is a silver bullet for every possible problem with releasing something.
You could release a well tested new feature with perfect ease and see nominal stats for hours after, and still have a problem you don’t see.
Maybe your new Calculator Widget 2000 works great for everyone except Big Enterprise Customer Bob. And maybe Bob doesn’t log on until 7 hours later because you’re in London and he’s in San Francisco. When he starts up the Widget, his mixture of data causes a pathological database behaviour. It’s a performance mistake in your code (that’s hard to find even with rigorous unit testing and manual QA).
Knock on effects follow like your task workers getting overloaded. So you have to saunter back in late on a Friday to fix it. You tiredly misdiagnose it as a usage spike, maybe some big new customer came on. You spin up 20 new worker nodes. Doesn’t help enough, tasks are executing too slowly. You figure it’s one of those mysterious intermittent performance issues in GKE. You begin to upgrade the cluster software in a desperate attempt, your thinking muddled by too many Friday dinner whiskys, to resolve it through the traditional GKE incantations. But cluster upgrades take forever and when you finally see the real problem and roll back your code it’s 2 am and your partner fell asleep on the couch and your kid wakes up at 6 am and God why did you deploy your well tested small new feature on a Friday?
My company has a rule of no deploys after 5pm (4pm on Friday) minus hot fixes and other special cases.
We have unit tests, integration tests, CI/CD, launch new systems/features via flagged code already in production, and we deploy 1-15 times a day to production.
Does this stuff prevent regressions, lower issues making into production, and make rolling back simple? Yes. Does it make it infallible? Not a chance.
Pull requests can be first in line to go out the next workday, no reason to eat into work/life balance and or company events.
I worked at a company that not only had Friday deploy freezes, but also the other days after 2pm, to avoid midweek deploys staying broken overnight. But some of our repos were set up to automatically deploy to production when code was merged into master, and didn’t have a develop branch; which meant that during half your work hours, you were not allowed to merge your pull requests unless it was for an emergency.
10 comments
[ 2.6 ms ] story [ 23.6 ms ] threadObviously it doesn't really work for companies that do continuous deployment, but for those that don't it provides great insight. Often you'll here an interviewer praise the fact the company doesn't do Friday deploys "because if it goes wrong that could eat in to our developer's weekends, and we like our developers so we'd never do that". That's the wrong answer. The right answer is "Of course, we trust our deploy process."
It seems over the top to have a blanket freeze but equally risk should always be considered and all deployments are not equal when it comes to risk.
Deploying Friday evening or during the weekend might actually be a very good time to do it if that corresponds to a 'quiet period' for your users.
What about the impact on your customers? That should be your priority.
Now, I agree with the premise of "trust your process" but if you've been into this field for a while, you know things will break.
Once a QA guy I worked with, came back from his weekend with a new test case/scenario that resulted in a critical issue right before we were ready to deploy. Not even CI/CD can help you with that.
However in a company where engineering's desire for a stable deploy pipeline with excellent visibility is matched up against business' insistence that all dev hours go into features and SEO improvements and every sprint issue regarding non-user-facing-improvements has to be fought for tooth and nail, "don't deploy on Friday" is a very reasonable compromise for avoiding weekend fires.
You could release a well tested new feature with perfect ease and see nominal stats for hours after, and still have a problem you don’t see.
Maybe your new Calculator Widget 2000 works great for everyone except Big Enterprise Customer Bob. And maybe Bob doesn’t log on until 7 hours later because you’re in London and he’s in San Francisco. When he starts up the Widget, his mixture of data causes a pathological database behaviour. It’s a performance mistake in your code (that’s hard to find even with rigorous unit testing and manual QA).
Knock on effects follow like your task workers getting overloaded. So you have to saunter back in late on a Friday to fix it. You tiredly misdiagnose it as a usage spike, maybe some big new customer came on. You spin up 20 new worker nodes. Doesn’t help enough, tasks are executing too slowly. You figure it’s one of those mysterious intermittent performance issues in GKE. You begin to upgrade the cluster software in a desperate attempt, your thinking muddled by too many Friday dinner whiskys, to resolve it through the traditional GKE incantations. But cluster upgrades take forever and when you finally see the real problem and roll back your code it’s 2 am and your partner fell asleep on the couch and your kid wakes up at 6 am and God why did you deploy your well tested small new feature on a Friday?
We have unit tests, integration tests, CI/CD, launch new systems/features via flagged code already in production, and we deploy 1-15 times a day to production.
Does this stuff prevent regressions, lower issues making into production, and make rolling back simple? Yes. Does it make it infallible? Not a chance.
Pull requests can be first in line to go out the next workday, no reason to eat into work/life balance and or company events.