I couldn't disagree more with what they say about feature branches.
> merging the feature branch with the main branch takes a lot of time
Why would that be the case? `git merge` is not a slow and expensive process. Is the problem the per-branch code review process that teams often engage in? They didn't even discuss how code is reviewed (feature demos are not code reviews).
> when the feature is done, merging branches back might lead to new bugs and instability
Feature branches should regularly be pulling in changes from the main branch. This seems like it would be a problem whenever your local main gets behind the upstream main, either way. Lots of teams will squash changes before merging feature branches, and run the test suite on it before it's merged.
> it is more difficult to do incremental refactoring
Smaller PRs and feature branches.
There have been a few posts lately espousing the removal of workflows that have evolved over time to solve real problems in development practices. To me, it seems like a lot of cowboy coders not wanting to be constrained by sustainable development methodologies.
I've seen feature branches attacked a lot and the common refrain is always the same. They live for weeks and then are hard to merge back in.
This is why you should be merging main back into your feature branch daily, so imo this is just a strawman.
That's assuming it's not reasonable to have smaller features. Sometimes it's not. You should strive for feature branches that live for a day or two, but if you MUST have it longer, just merge back in.
> tests are great in the long run, but can slow you down in the short run.
This is just silly, especially given you mention freezing releases and having an entire QA team go over the product. It would be much faster to have no QA team, no freezing of releases and instead have a robust test infrastructure.
It’s fine that this set of rules works for your company, you like them and are familiar with them, but that’s all this list is.
Why is everyone in this thread forgetting what tests actually do?
Tests force you to write better code. Code that is easier to change. Code that is more robust.
You don't need 100% coverage or perfect tests to gain this benefit.
A good test suite enables you to make large refractors safely and quickly.
The most scary refractors I've ever had to do are ones that involve large changes to code that has no tests.
Often those refractors that allow company level pivots require adding huge amounts of tests to a legacy code base before I can even get started.
Adding tests to code that has none is the worst. The code is coupled all over the shop and requires 100 times more mocking and stubbing to get under tests when you add this stuff retroactively.
Tests speed up development velocity and improve your codebase.
Just don't get stupid about it and you won't negatively impact startup performance.
I generally feel the highest value tests are at the lowest level (code with few dependencies) as well as end-to-end integration tests. It gets a little murkier in middle layers since alternate implementations of dependencies (aka mocks) need to be created, or one has to deal with complex setup of dependencies or in some cases re-testing the same logic that has already been tested at a lower level.
I'm not suggesting middle layer testing is not valuable, shouldn't be done, or to never use mocks, etc., it is just much more difficult and more creativity is required to add value in this area. For example, in some situations middle layer testing requires the creation of various kinds of utilities and frameworks in order to find a reasonable benefit vs effort sweet spot. Dogmatic approaches (such as mocking all dependencies in all situations) feel like they are productive at first but can end up being costly as each mock is an alternate implementation of an interface and doesn't necessary capture the behaviour of the concrete implementation.
I hardly ever do unit testing. It's almost always integration testing with code that uses real implementations against a testing db. This is only possible because my code is decoupled in the right places to enable this sort of no mock style testing.
I don't strive for 100% coverage and I hardly ever do TDD unless I'm working on something hard to solve.
Tests give me confidence and results in less time repeating the feedback loop. I might have a slower "coding speed" but I have fewer regressions during refactoring. Less bugs to fix while iterating. And less back and forth with stakeholders to ship something. Overall I'm way faster with tests than without.
1. Feel free to write tests that help you develop your thought or solution. Not all of those should be checked in
2. Ask yourself: “What test do you need to write in order to protect this code from your future self next year when you have forgotten everything about this?”
Omg...do not like where this conversation is going. Handing people untested software is ridiculous. At that point we might as well have cooks dip thier hands in raw chicken before making your salad or encourage machinists to dawn long flowing scarves.
People who don't test don't test because they don't know how. They are hacks.
> People who don't test don't test because they don't know how. They are hacks.
Or maybe they're just doing a different trade off. Do you unit test all of your shell scripts and jupyter notebooks?
If not, why is that? Is it that some code is either obvious enough or doesn't warrant having tests for it?
There's a spectrum of testing, from zero tests (which is fine for throwaway code), to testing a few key components and having end to end tests for the rest, to having 100% branch coverage. One should write enough tests that they're confident about the quality of the software that they're shipping, and no more, in my opinion.
I work at an early stage startup. We evaluated testing vs velocity. Our conclusion was almost universally:
* BE tests are easy to write, catch lots of important bugs, and generally speed up development.
* FE tests are painful to write, rarely catch meaningful issues, and don't really save any effort - devs still need to click through a UI to make it's correct.
* E2E tests were even more painful to write. Likely caught more than FE tests, but not enough to justify their even more insane overhead.
----
Typescript does wonders for FE test coverage. The bugs we do run into tend to be silly oversights (empty arrays, FE <> BE type mismatches, etc). Annoying, but quick fixes. Not worth a 30% overhead on every feature to save 5% occassionally.
I'll give you a case where this is true if you're prototyping something new and need to keep changing fundamental things like the data model.
I have nearly 100% unit test coverage for a couple of the base libraries I've created, but yeah, writing tests was slowing me down significantly for the rest.
Once I have something where chunks of the application aren't being scrapped every other week, I'll start adding unit tests to it too.
I think the author is just saying not to have draconian testing requirements for every PR like some companies do.
But really, I can see it. Firstly, a lot of engineers suck are writing tests. White box unit tests with interfaces and mocking everywhere tightly couple code blocks together. Integration tests are nice but take a lot of effort to get right while also keeping them performant. When (not if, remember we're talking about an early stage company) the codebase and/or the database schema changes dramatically it will often break the tests which is just another hangup.
> Integration tests are nice but take a lot of effort to get right while also keeping them performant.
In my pretty long experience it is a product issue, not tests'. And that is a kind of issue which sets the tone for that product development for years ahead.
>the codebase and/or the database schema changes dramatically it will often break the tests
I think it is silly to use this one sentence without a context and start arguing. The whole paragraph in the article makes sense, gives context and advice is reasonable.
A reasonable middle ground is to have e2e smoke tests just so that you have the confidence that the most business critical flows are ok even as code changes. Doesn’t require that much effort but 80/20 rule style gets you big bang for your buck.
In my experience, tests speed you up in the short-run too. I've run this experiment myself at my own company within different teams. Code that lacks tests has the most bugs, the most QA returns, the most churn, etc....we end up spending (read: WASTING) much more time and money fixing broken code that has no tests than any other code in our entire codebase.
In my experience, during the early days of a product, things change so rapidly that tests are extremely difficult to maintain. On the other hand, if you're not going to have tests, you need to be able to respond to, diagnose, and deploy fixes for bugs within 30 minutes.
Yeah this is by no means one size fits all. The appropriate mix of QA and automated testing really depends on what kind of product you're building but I can't think of any use cases where you want no automation whatsoever.
Something that's missing from this list that I find extremely useful:
An unstable environment with actual users doing real things with it. They could be internal, they could be advisors, they could be customers who are also investors or are willing to be testers for a discount.
That works really well if your code is designed to fail loudly (validations, assertions, etc...) and you have a culture of fixing issues immediately.
This is basically the "load up on tech debt early" play. If you grow you (presumably) have money to fix the debt, if you fail (at business) the code debt is zero (dollars)
the problem with this play is that the time never really arrives. and bad architecture begets more bad architecture until you're left trying to sell a rewrite, which nobody thinks is a good idea...or sell 1-2 yrs of painstaking incremental rewrite with additional test development.
the latter is the only hope, but practically no one is willing to commit to it
now when I talk to people about making their product be something that development can be done on again - I try to figure out how serious they are. almost never.
tl;dr figure out what you want before committing all your resources.
It might be just me and a terrible string of bad luck bad I've seen so many projects that were kickstarted preemptively due to company politics or just management wanting to secure additional funding.
By the time (usually between 3 to 9 months) business stakeholders came to an agreement on what they actually wanted the team developed so much technical debt that it's virtually impossible to untangle from all bad decisions for the rest of the development cycle.
That's why I'm a huge advocate of starting simple, with a monolithic approach, to tread the water and have enough time to figure out the true objectives of the project first.
The best versions of this I've seen have QA folks embedded in the service/product team to own the release pipeline, and their job isn’t performing tests, but writing automated ones, directly pairing with developers, analysts, SREs to do so. It's the antithesis of the all-too-common scenario of QA being isolated as a separate team.
The primary goal is obviously to produce useful and targeted feedback as early as possible, but more strategically it elevates the status of QA, making it a gateway job to developer, SRE, or product management. In some organisations the QA function is perceived as a dogsbody role, and a reframing helps to flatten the pecking order and improve access to the talent pool.
Before we all went remote work, I'd always recommend adjacent seating for quality folks, to exploit the natural proximity bias. Alas, I'm still not sure what that looks like today.
Our team actually dropped our FE testing. After doing it for a year, we basically came to the conclusion it wasn't catching bugs we cared about while costing us loads of time. We still test on the BE.
On the FE, we noticed:
* Most of the bugs were in UX/UI that are very hard to capture in a test suite.
* CSS/UI bugs are better handled with a good UI library. We have an _amazing_ UI dev that's built a really stable UI library for the team.
* It took forever to write FE tests.
* It took even longer to run FE tests.
* Even well-written FE tests miss a lot of subtle issues.
* We rarely had "random" bugs. UI bugs were almost always in the code we changed. Easily detect with quick click testing.
I have looked in to front end unit tests a bunch and never seen any way to do them that makes sense. Jest and the lot just don’t seem that useful.
What _does_ work is front end integration tests with cypress/selenium. You just write some basic tests that click through the flows and see you got to the end. This catches issues all the time where the api response or something has changed unexpectedly and the UI gets stuck half way on an error.
It’s also super simple, it’s 10 likes of looking up elements by ID and running .click(). They hardly ever break from normal UI changes and prove that the app is mostly working as intended.
Switching to TypeScript is also a big one. It’s almost like half a test system given to you for free.
I appreciate your position but I could not disagree with you more. React and other frameworks allow for components to be unit tested very easily, with good control over the inputs. This gives confidence in each component which can then be added to with a (very small) set of end to end tests.
I would also state that "10 likes of looking up elements by ID and running .click()" is a big part of why e2e tests are flaky and hard to maintain. Good e2e tests require good engineering.
Verfying functionality is not even the primary way tests speed things up, in my experience. What really speeds development up is high quality code, and code written to be testable is usually of higher quality than code that is not.
In a similar manner, putting legacy code under test tends to remove a bunch of bugs; not because the tests find them, but because when I refactor the code to be testable, the bugs in the code become much more obvious.
I’ve led a fairly large product org in recent years, and frequent demos of deployed software is the number one way to ensure you’re shipping software frequently and consistently
On my team it's used as a false deadline, causing us to ship low quality code without sufficient tests, ultimately trading long term delivery dates for short term ones.
This is a big one. It seems the industry (at least to my awareness) has almost totally rejected Gitflow for trunk based development in the last few years. It’s a beautiful idea in theory, but a complete headache in reality.
> Use feature flags
Be careful here. Adding feaure flags means you now effectively have N^N builds (where N is the number of flags) in production, each with their own possible quirks and integration issues. Keep them to an absolute minimum, and remove them as soon as possible.
There are a lot of good practices there and I commend the author for sharing. I disagree with code freezes + manual QA verification -- why just dip your toes into CICD? -- and the advice about writing tests. "We don't have time to test now" results in a vicious circle.
It's an interesting approach, but it seems like by the time PR 1 is reviewed, much of it could be out of date as you continuously refactor along the way.
Also what happens if during the review of PR #1, the reviewer suggests something profound that would fundamentally change the direction of future commits?
Agreed. I'm not using feature branches because it's fun, I'm doing it because it reduces the time wasted on figuring out how git/our product works. Stay up to date and you won't have an issue.
>> If you can't get customers, what's the point of having a sophisticated codebase with tests, etc?
And in fact every minute spent on writing and running tests (and there's always ALOT) and CI/CD is wasted, wastes money and time, leading to longer development time until the software can be put in front of users.
Therefore CI/CD and tests and all their associated processes are a red flag and increases the likelihood a startup will fail to bring its software to market.
Developers who have always worked in well organised environments have really deep rooted trouble getting their head around this way of thinking. If you say to an experienced developer "no tests! no CI/CD!", it's likely they will immediately decide you are an unprofessional hack.
I met a founder with this mentality--but the problem was, the company was no longer a start-up, he had dozens of products, many (unhappy) customers, and zero development process that wasn't "writing code". He still believed every minute you spent on writing something that was not directly shipping to customers was his money being wasted. I had to argue with him to even get source control set up (he didn't agree, and I just did it in secret at night anyway). CI/CD?? Fuggetaboutit. Releases were pulled directly off the PC of any engineer who managed to get the software built without errors that day. Yes, they were usually debug builds with symbols, and never reproducible. Needless to say, it was a clown show.
I hear ya; you're not off on most of this. However if you're building a web service, Vercel (and its kin) make CI/CD ridiculously easy. Easy as in like 5 minutes easy, and it's worth the 5 minutes for the all the benefits.
>> Easy as in like 5 minutes easy, and it's worth the 5 minutes for the all the benefits.
The sum total of time spent on CI/CD and tests is not the time it takes to get it started, it's the time spent executing the tests/CI/CD into the future, thinking about the tests/CI/CD, problem solving the tests/CI/CD.
All these things require ongoing time which takes time away from developing the application.
Thinking about the tests may be an ongoing cost, but the whole point of CI/CD is that I never have to think about certain things again. That's also the idea behind the tests, but there is admittedly more work keeping the tests up to date than the CI/CD pipeline.
You stopped telling the story at the most interesting part.
At my last startup, I wrote totally throwaway code on all our initial prototypes. Once we had something that looked like it would hit product-market fit, I hired a team, we threw away the shitty codebase and started fresh.
If you don't throw it all away after making the kind of mess you describe, then you are going into the company's most crucial time with a boat anchor around your ankle and the water level rising. I'm very glad we did, as starting fresh with tests, CI/CD, and other quality practices let us be extremely responsive to my cofounder's needs and the things we learned from every release.
This is why an experienced developer is sometimes the wrong person for the job to build a startups first version, unless they are able to fully embrace the mindset of prioritising speed over everything.
Beware of experienced developers who get the job of writing a startups first code and who start talking about tests/CI/CD/feature branches and all that stuff.
Avoid security holes and take backups, that's all that matters and anything else should be actively avoided.
Again, you seem to be stopping the story at the wrong point. The initial job is to find out whether the project is worth investing in. 9 out of 10 startups fail, they say; if you're in one of the 9, you need to know that ASAP. If you can tell that with throwaway code, great! You've saved a shit-ton of money.
As an example, I did a startup circa 2010. We raised money on an idea that VCs were very excited about. So much so that we found out later that other VCs had invested in similar notions. We spent a few months building disposable prototypes to test the value prop. It turned out that users hated the idea no matter what we tried. So we killed it after spending maybe $100k and did something else. Job done! But the other companies persevered, building full apps, and each burned millions before they realized there was no future in it.
So in that case, yes, you don't want to invest a lot. But a few ideas later we had something that looked to have real product market fit. At that point, we threw away the code and started fresh with solid tests, CD, and other quality investments. Why? Because the job we were trying to do had changed. It had gone from hypothesis testing to actually serving users and building a real product. If you want to get that job done well, you can't afford to have a garbage code base.
...if only I had a cent for every time I've had this conversation with an engineering manager: Manager: "Just live with the mess for now. We'll fix it later when we have money." Later: "Can we fix the mess now?" Manager: "No. We're making too much money now. Blah, blah, blah.
Opportunity cost."
We really did throw it away. And when I'm an engineering manager, this is my big test for product managers. Initially I give them a low-limit credit card for technical debt. If they pay it off when they promise, I give them a bigger limit. If not, I take the credit card away and we keep a consistently low level of technical debt.
And I should add that more startup and new-product teams should be bold about throwing away code. Startups need to be optimized for learning. Each release should tell you something about your users, your market, your ideas. But doing that means that some of the code will have been written before you know what you're doing. That code either needs to be refactored or reverted. The latter feels scarier but is often cheaper.
That is ludicrously bad advice. Setting up a straightforward CI/CD pipeline can take a couple hours and it instantly provides huge productivity benefits.
I worked for a startup that didn't have an automated deploy pipeline and didn't have any tests. It was a complete shit show from day 1, and I will never work for that chaotic of an environment again.
I am certainly not for over-engineering early in a company's lifecycle, but there are simple, "table stakes" things like use of SCM, automated builds and CI/CD pipeline. If I don't see those I assume your engineering team doesn't know how to build software fast.
> The number of hours I have seen developers spend on things related to CI/CD is vast.
Sure, same as everything else that is overengineered. A simple, no-frills GitHub Action (or equivalent) should not take longer than a few hours to implement. If you need more time than that, it means that you have an overly complicated deployment process, which is a time sink when done manually anyway.
Your software should meet the minimums standards required by your target user.
That's the "viable" part of Minimum Viable Product.
In a similar vein, you cannot release a new antivirus product that does not detect viruses, or security product that does not make things more secure. Software development has (or should have) defined priorities - and if accurate financial transaction processing is a base requirement then that's the minimum viable product you need to deliver. And if it takes tests to do that, then that's what you need.
Yeah devs also don't need to use VCS, and instead of hosting it on servers just host it from a laptop using ngrok. Better yet just get rid of the code entirely and manually perform whatever tasks your users need.
CI/CD takes like 3 hours to set up and saves you hundreds of hours in deployment time. It delivers new features to your customers faster, lets you ship more frequently because you know you can roll forward quickly or revert when you mess up. Any startup founder who tried to tell their devs not to bother with CI/CD (if it's a real product, not just a prototype or proof of concept) probably doesn't know what they're doing.
Being a startup doesn't mean your tech has to be garbage. So many miserable devs hoping that "after series A" they'll get to refactor that spaghetti code mess (they won't) and seeing their velocity slow to a crawl because they can't make a change without worrying the whole thing is gonna break.
1. you are stating an opinion as fact, which is rude. Other people have facts that disagree with yours.
2. you are wildly unclear about your preconditions. I am pretty sure you are talking about the initial stages developing an MVP for a startup. Other comments are probably referring to the startup growth period following the MVP.[1]
3. You might be right in some circumstances. But your assumption elsewhere in comments that CI causes over-engineering is incorrect: any team that has an over-engineering problem will generally be in trouble, which is not caused by CI.
4. You are replying to many comments dismissively, almost attacking with your “facts”. If your argument is strong, others will answer for you. “ Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.” — https://news.ycombinator.com/newsguidelines.html
[1] I’m using this definition for startup http://www.paulgraham.com/growth.html — personally I dislike PG’s definition of a startup (paraphrased: designed for fast growth). Should a business that is trying-but-failing-to-grow not be a startup? A business designed for slow growth but large market is not a startup? PG gives Google as an example of a startup, yet that contradicts his own essay: didn’t Google have a lot of luck, and it isn’t obvious to me that Google was designed for growth, more that the opportunity knocked and Google jumped on that opportunity, successfully. Some of PG’s definition feels tautological; 20/20 hindsight.
>> 1. you are stating an opinion as fact. Other people have facts that disagree with yours.
Agreed. I have made an error there.
>> 4. You are replying to many comments dismissively, almost attacking with your “facts”. If your argument is strong, others will answer for you. “ Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.” — https://news.ycombinator.com/newsguidelines.html
Agreed. I have made an error there. I should choose a better tone. I do work hard to get the tone right but not successful today obviously. Apologies if this has offended anyone. I usually delete things that I feel are too negative, adversarial, preachy or otherwise not the right tone.
>> 2. you are wildly unclear about your preconditions. I am pretty sure you are talking about the initial stages developing an MVP for a startup. Other comments are probably referring to the startup growth period following the MVP.[1]
The comments I make are about a company pre product/market fit, doing the first build of its software before opening the doors to customers. I do agree that the startup period continues beyond this perhaps for a number of years, at which time development priorities might change.
For reference, in the link PG says "A startup is a company designed to grow fast.". This seems overly narrow because it seems self evident that there is lots of startups that have different objectives to fast growth.
My definition for a startup is more like Steve Blank's: “A startup is a temporary organization designed to search for a repeatable and scalable business model.”
Chaos has a cost too. You need to weigh that against the cost of testing. And both chaos and testing cost exist on a very broad continuum. For very trivial prototypes, spending even a few hours on tests obviously outweighs any benefits. And 100% unit testing coverage never pays off even on larger projects. However a few hours of setting up CI/CD with linting, type checking, required reviews, automated deployments, and yes some sanity checking integration tests, can provide massive velocity benefits that pay back the investment in a few days and are pure velocity windfall thereafter.
I think you (a) have been burned (and I'm not denying testing can be a waste of time) and (b) not being a dev yourself you don't fully understand that this isn't a black and white issue or that you are actually paying a massive velocity cost by doing zero testing. Your POV is akin to advising people to not drink water on a hot, strenuous hike because "every time you lift your arm to drink, you are wasting energy that could go to your legs". Uh yeah, in both cases it's a little more complicated than that. Definitely don't waste an hour making a gourmet meal (and yes I get sometimes devs do this), but there are a spectrum of options and there is a sweet spot. How about "drink some gatorade and eat a power bar without slowing down"? Getting that balance right isn't trivial but it is possible and the payoff is huge. Which goes to the other point you made about not hiring expert devs, man, you've been hiring the wrong experts if you think fresh, unsupervised bootcampers are a better choice at any stage other than "can you make me a framer prototype over the weekend?".
This is counterintuitive because the more productive your development team is the more security holes they will leave behind. The thing is how to keep the productivity while increasing the security.
Fair point, but going through the comments, I'm not trying to sell anything. One is reframing the suggested question and sharing a consolidated list of resources. The other two are on a free tool (not suggested before) to solve the problem stated, so the aim was to expand the options. But point taken ;)
I think this depends on the stage, but I don't think for us deploying weekly instead of 4x per week (M-Th). Would help.
We also found feature flags slowed us down a lot in the early stages. I feel these should only be done for some features (e.g. an explicit AB test, a large feature with a lot of technical risk). Feature flags create a ton of complexity, and are often not worth it.
Agreed on both counts. At my last startup, there was a release every few hours. We used feature flags for anything where the code's future was uncertain, especially experiments. But once the experiment was resolved, we mercilessly removed the flag, refactored the code to reflect what we'd learned, and added tests.
Deploying 4x is better, but not every team can do it without sacrificing quality.
That essay was written based on 3 three pre Series A companies, which didn't have right processes in place and was struggling to ship fast enough. Weekly deploys is a first step. It's difficult to impossible to deploy more often if you don't have good tests coverage.
There's some good advice here, but this is ridiculous:
> But, on another side tests slows you down as you build features. When your team writes tests they need 2x more time to release any feature.
Tests are an investment in fast future development. Last I did a startup, we did not write tests for anything throwaway or experimental. But once we decided some chunk of code was a keeper (e.g., experiment had a positive result), we wrote tests.
With just a team of 6, we were releasing a few times a day, not just once a week. And we had no "QA freeze" nonsense, because our tests were solid enough that we didn't need manual QA to release.
That said, I hired for people who were good at writing tests, and we did pair programming with frequent pair rotation, which gave us a big quality boost and helped keep our test suite lean and effective. I can believe that some startup teams have low enough levels of testing skill that "they need 2x more time to release any feature".
Sure, and that's why I explicitly said "experiment had a positive result". Before you decide a particular chunk of code is a keeper, you should have data that gives you confidence.
Before you have signs of product-market fit, I think people should write throwaway code and then throw it away. Once you have something worth investing in, I think it's time to invest. Having high-quality code in areas where you're confident it's valuable enables rapid experimentation in the areas where you're not sure yet.
I’m having a hard time reconciling “use CI/CD” with “deploy weekly” and “freezing code before release” is certainly not in service of “developing weirdly fast”.
My company actually does this - we just found that a) we wanted to be able to tell our clients about new features instead of them “stumbling on them,” which makes more sense when you’re dealing with low numbers of high value high touch clients; and b) we wanted to have greater confidence that our code was working properly in the absence of a thorough test suite. Feature flags could help with a) and a good test suite would help with b) though.
To be clear I’m not saying it’s “bad”. Just that the article purports to be about how develop things fast. And purposefully slowing down your releases is not that
Respectfully, that is worse than just deploying once a week (which is totally fine btw. Lots of reasons to do that)
You’re still waiting to release features that are done and turning on a whole week’s worth of features at one time, but it’s even worse because you spent a bunch of extra time building it to have feature flags.
I think you're correct only in the scenario where the feature-flags are binary on/off switches.
The modern "feature flag" is more of a knob that can be slowly turned up, along with ideally a way to send shadow traffic through the feature even before any % of real traffic is allowed through.
In that proper implementation of feature-flags, the overall benefits (easy rollback, graduated rollout, selective rollout based on user characteristics, better ability to manage scale) outweigh the rather-trivial engineering cost of wrapping a few if-clauses around the feature.
That does not sound much like a “weekly release” though
> rather-trivial engineering cost
I personally disagree with this. To implement feature flags your engineers have to reason about all the places where the feature needs to be on/off, implement those flags, write and maintain test cases for both the on and off case, rollout the code, then go back and refactor everything to remove the flags afterwards.
It’s not insurmountable. But it’s not trivial in my experience.
> That does not sound much like a “weekly release” though
I respectfully disagree, because I'd say that a "release" ought to be defined from the viewpoint of a given individual user. From that person's perspective, they are indeed seeing things change only weekly, and the details of how that's done (e.g., feature flags vs giant deployments) are mostly irrelevant to them.
> It’s not insurmountable. But it’s not trivial in my experience.
You are right. Honestly, I'm so used to having to sell the idea that I just reflexively wrote it that way.
Thank you for listing out the different pieces. You're absolutely right there is a lot of extra "essential complexity" that gets added in. Even if the "actual" coding work is "just a few" extra if-clauses and tests (which is how I'd sell it ;), everyone has to hold a lot more in their heads.
Personally, though, I would still insist that it's way better than the essential complexity that gets added everywhere by coupling "releases" with "deployments." In my experience, once an org lets product-release-tempo dictate engineering-velocity-tempo, the bureaucrats essentially have a chokehold on all engineering decisions.
The organization slowly grinds to a halt; high-value customers start getting picked off by nimble, faster-moving startups; and the circle of life continues.
Feature flags are IMHO a key technique for a startup-that's-getting-bigger to hold off this particular mechanism of decline. Non-technical stakeholders at all levels can be easily waved off with a simple remark of, "Well, we'll feature-flag it." And engineers can stay much more empowered, even as the company grows much bigger.
Monorepos are unreasonably effective for a product that you can guarantee won't scale to silly proportions. They also provide you some great capabilities for introducing new team members to your solution, bringing aboard contractors if something breaks or you face some regulatory issues later on, etc.
The stuff under "Freeze the code before release", such as a code freeze where devs can't make any changes, seems to be the opposite of the article's goal. Your release should be a specific commit ID, not a branch name, so it doesn't matter if people continue to develop and push changes. If you need to make changes specific to the release, use a release branch. No need to stop devs from moving on.
These are at odds with each other. In my previous role at a startup we deployed on average twice per day per developer. The automation necessary to make this hands off and reliable wasn’t actually that much. It rarely caused issues.
We're doing all of these things and we're weirdly very slow. But his first paragraph mentioned motivation, which I think no one has: no pay increases, people quitting (and not just quietly), customers upset, no stock options and just all around no one likes anyone else at the company any more.
I agree with most of these, and vehemently disagree with a couple. I think we move pretty damn fast, we average 5 commits per dev per day.
I very much agree with break features into many small pull requests. We merge constantly even (especially) if the code path won't be hit yet. Just get it into production. Makes the code review faster, the complexity lower, and lets you focus on one thing at a time.
I think no tests makes you slower. We paid a bit of cost to start writing tests, but now we can deploy 10 times a day because we know the mission critical stuff is working. It's more about testing the right stuff. Decide what can break, and leave it untested. What can't break should be covered by tests. We do stuff with money, all that is robustly tested. The UI isn't.
Code freezing / weekly releasing is counter to CI/CD. Just make your deployment take less than 10 minutes, and ship code constantly.
Agree with this strongly. Weekly releases reduce efficiency and lead to more complex production issues as you work to unpack which part of the batch of things you just deployed is causing a problem.
Use feature flags to decouple Deployment from Release and you’re fine.
Feature flags are good but I hate how they enable weird customer specific workflows. Imo the mindset should be that all feature flags will eventually be removed when testing is finished.
But management sees the capability and says add a flag for customer Y that changes how this page works.
Agreed. IMO the best approach is a flag that can let you enable for specific customers (alpha or internal sample customers), all beta customers and then all customers.
And after it’s enabled for everybody, remove the flag.
> Just make your deployment take less than 10 minutes
This is great until it's not. We're starting to hit this tipping point on our code-base.
The challenge for us is it'd likely take an engineer a week or two to optimize our test suites. That's enough time for them to ship meaningful features.
Instead, we do occasional freezes for important releases. Not sustainable, but it's a good enough bridge.
The slowness in my tests comes not from having too few runners (we just dynamically spin up a runner for each commit, and it normally takes only a few seconds for that), but from serialization in the CI pipeline. You cannot run tests until the code has been built; no extra runner can change that.
I need dev time to make the builds understand their inputs, and then cache more aggressively than we are, or something. (But I lack the dev time.)
pipeline time is something that I spend a non trivial amount of effort on.
caching and promoting images is an absolute must. caching images has it's downsides, too though. it can be a longer time period between when something in the dockerfile breaks and when it's noticed.
generally speaking, my pipeline is generally 10-40% build time, 50-60% integration & system test time, 10% deployment time
The system tests that're run deploy the entire dependency graph of services that my project depends on, which can take a bit to settle, has a high chance of transient failures relative to other parts of the pipeline, and notably has a fixed test run time.
That last bit really raises the lower bound for how long the pipeline as a whole takes -- number of runners doesn't matter.
The pipeline generally takes between 25-45 minutes, and runs on each push and is triggered by changes to dependencies to ensure backwards compatibility.
I wouldn't do anything to make it take longer than it does now, and it can be frustrating. but the pipeline will catch things that would otherwise slip through the cracks.
Same thing goes here. It works great until it doesn't.
E.g. we went with that route with GH Actions, until our tests didn't fit into the memory of the provided machine sizes anymore (shouldn't be a problem anymore as GH provides options for bigger runners now). So we had to spend 1-2 days on setting up a self-hosted runner in a way that it wouldn't be too heavy in maintenance, and then probably another 1-2 days of rework over the next few weeks to iron out the kinks.
I mean if this works for you, great, but it sounds like a dangerous dynamic to aim for.
The occasional freezes mean future tests written do not need to be optimised (there will be a freeze for important releases anyway), so the unoptimised-tests problem will grow worse the longer you use this strategy, and the cost to rectify it will grow, until your release frequency collapses on itself and everyone complains that "there's no solution to this problem".
These are the sorts of things I think it's very important to nip in the bud. Spend the two weeks now, when it only takes two weeks! Much better than putting everything on hold for six months to fix it four years from now when it becomes untenable.
>> The challenge for us is it'd likely take an engineer a week or two to optimize our test suites. That's enough time for them to ship meaningful features.
That does seem like a possibly worthwhile choice though? A "stop chopping the tree to sharpen the axe" moment?
That might make sense if it'll make or break you. If successful you can afford to fix it later.
But, maybe spend a day on it. I've been to places where the build took hours and there were obvious 30min solutions to fix it (which I did, in some cases).
Consider it an investment. 2 weeks now gives you how many more features over the next six months due to increased dev velocity?
This might be tied in to the language ecosystem of tooling but after many jobs, I’m yet to see bad tests in ruby. Perhaps there are some CPU cycles wasted on the CI for not so useful tests, but it’s easily paid back by the mountains of useful tests.
A simple example of a bad test is one enforces an incorrect behaviour (i.e. 1 + 1 == 3). I can't see how a language could improve this situation. Is it possible it is the community as opposed to the language itself?
I’m not sure how this really slows you down though. You see the app code is wrong, it breaks a test so you update the test to the new expected result.
The benefit is that the tests highlight what changed in the app. So when you update a library and 1+1 changes, it’s an alarm bell to investigate further because you didn’t expect this to happen.
This is an example the test/hash antipattern. Here, the tests approach a hash of the source code. When some code changes, the hash no longer matches which fails the test. Great, this is letting us know that something as changed. After examining the changes and decide that they are correct, we update the hash and now the all the tests are green again. Our code base has 100% coverage.
Bad tests can include things that timeout due to slow resources, unnecessarily depend on external resources, and stuff like that. It’s a waste of time to debug flaky tests that didn’t ever need to be flaky.
Yeah I can see that being a problem. It's always been understood that tests should never reach out to the outside world in the teams I have worked in. And there is a wealth of tooling to mock responses.
Longtime rails dev here, the biggest problem I keep running into is testing at inappropriate levels. For example, writing two cucumber (or system) tests to confirm conditional rendering or search filters or some such. These are more appropriate as a view or model test (respectively) and running them naively through the full stack is a huge amount of overhead waste.
At an early stage startup with a small app it might not be as big a problem, but where I'm at right now the app is over 10 years old and our test suite with many many parallel workers is still well over an hour cycle time to get that sanity feedback on a commit, and it's painful.
They had me until they said deploy weekly. Deployments shouldn't be an event. Deployments should happen constantly. Get your stuff into production immediately. All the other practices up to that point support that.
As someone working on a hardware project... hahahahah
We manage daily, but even that is a miracle. If you're building enterprise CRUD app #310979707 or SaaS project #2009879, sure, deploy constantly. But a very significant number of projects simply cannot do that.
Deployments depend on what you're deploying. A web app, maybe that's ok. A desktop application with automatic updates can be updated daily at most. Other products without automatic updates are probably better served with longer release cycle.
I'm not the original asker, but evaluating programmers based upon the number of commits per day is a lot like evaluating them based on the number of lines of code written per day -- it's a measure of activity, rather than a measure of productivity, and those often aren't the same thing.
(I'm not trying to imply that that's a mistake you've made; it's just something that it's easy to forget as a manager of programmers, and is maybe worth pointing out explicitly; you need a way to measure the team's productivity, but it's very hard to find a good, measurable proxy for 'productivity' which isn't eventually going to end up being gamed by the whole team!)
I think that's fair. But I would say the article is about activity (shipping features fast) and so as a measure of activity commits per day is a decent proxy. Obviously the devil is in the details.
I would not judge an individual developer's productivity on commits per day, agree that would be foolish.
I feel like I didn't give enough context (I've written that essay) to the kind of projects that was used to test what is written.
1. Pre Series A companies, which didn't have strong engineering team from start
and now struggle to ship.
2. Idea/Pre Seed companies, which just want to ship fast and raise funds. That kind of founder usually have 'friends & fools' budget.
1. First kind of projects usually don't have enough tests coverage to deploy safely more often. Typically I would set a goal to deploy twice a week in 6-12 months and build a plan of how to achieve that.
2. Usually I advice to not write lots of tests for the first ~6 months to focus on shipping faster. In my experience products often change dramatically during this period. But a) there are exceptions. Some domains (e.x. finance) are critical to write more tests. b) tests infrastructure are always set as part of CI/CD process, so engineers can decide and write them for area of high importance.
Anyone talking about code coverage percentages is already making a mistake. percentage of coverage isn't nearly as important as WHAT is covered, and it's not a good proxy for it either.
Not writing tests early on simply backloads the effort to the point of where deploying anything is a crap shoot.
Most of the other things I can agree with but tests is literally 101 for my teams
I have worked for a lot of startups, and the best ones have deployed as often as possible. One company (which grew massively and is now public) allowed developers to push code whenever they wanted. Once your code was merged into master and the required tests had passed, you could push to staging and then production whenever you wanted. However, you were responsible for making sure that your push didn't break anything. Developers took a lot of responsibility, and when something broke, the last guy who pushed usually fixed it really fast (and if not someone else did). Releases went out several times a day.
At my current job (fast-growing mid-stage startup), any developer can push when they want, but it works a bit differently. Once the release is staged, everyone who has a commit on staging is responsible for verifying their feature. Then the developer goes through a "release checklist," which involves finding a buddy and jointly running through a list of manual tests to validate key features. After that, they can push. They probably push a few times a week.
I feel like code freeze/manual QA/weekly push is something of an anti-pattern. Developers are disconnected from the process of releasing their code, and take less ownership. While in practice manual QA should lead to finding more bugs, in practice it can lead to more defects.
210 comments
[ 193 ms ] story [ 2420 ms ] thread> merging the feature branch with the main branch takes a lot of time
Why would that be the case? `git merge` is not a slow and expensive process. Is the problem the per-branch code review process that teams often engage in? They didn't even discuss how code is reviewed (feature demos are not code reviews).
> when the feature is done, merging branches back might lead to new bugs and instability
Feature branches should regularly be pulling in changes from the main branch. This seems like it would be a problem whenever your local main gets behind the upstream main, either way. Lots of teams will squash changes before merging feature branches, and run the test suite on it before it's merged.
> it is more difficult to do incremental refactoring
Smaller PRs and feature branches.
There have been a few posts lately espousing the removal of workflows that have evolved over time to solve real problems in development practices. To me, it seems like a lot of cowboy coders not wanting to be constrained by sustainable development methodologies.
This is why you should be merging main back into your feature branch daily, so imo this is just a strawman.
That's assuming it's not reasonable to have smaller features. Sometimes it's not. You should strive for feature branches that live for a day or two, but if you MUST have it longer, just merge back in.
This is just silly, especially given you mention freezing releases and having an entire QA team go over the product. It would be much faster to have no QA team, no freezing of releases and instead have a robust test infrastructure.
It’s fine that this set of rules works for your company, you like them and are familiar with them, but that’s all this list is.
At that point most products are basic enough for developers to test the product themselves.
And if the codebase is undergoing lots of churn and heavy refactoring then tests unquestionably will slow you down.
Maybe it is better to only begin automated tests when you reach product-market-fit.
Tests force you to write better code. Code that is easier to change. Code that is more robust.
You don't need 100% coverage or perfect tests to gain this benefit.
A good test suite enables you to make large refractors safely and quickly.
The most scary refractors I've ever had to do are ones that involve large changes to code that has no tests.
Often those refractors that allow company level pivots require adding huge amounts of tests to a legacy code base before I can even get started.
Adding tests to code that has none is the worst. The code is coupled all over the shop and requires 100 times more mocking and stubbing to get under tests when you add this stuff retroactively.
Tests speed up development velocity and improve your codebase.
Just don't get stupid about it and you won't negatively impact startup performance.
I'm not suggesting middle layer testing is not valuable, shouldn't be done, or to never use mocks, etc., it is just much more difficult and more creativity is required to add value in this area. For example, in some situations middle layer testing requires the creation of various kinds of utilities and frameworks in order to find a reasonable benefit vs effort sweet spot. Dogmatic approaches (such as mocking all dependencies in all situations) feel like they are productive at first but can end up being costly as each mock is an alternate implementation of an interface and doesn't necessary capture the behaviour of the concrete implementation.
I don't strive for 100% coverage and I hardly ever do TDD unless I'm working on something hard to solve.
Tests give me confidence and results in less time repeating the feedback loop. I might have a slower "coding speed" but I have fewer regressions during refactoring. Less bugs to fix while iterating. And less back and forth with stakeholders to ship something. Overall I'm way faster with tests than without.
1. Feel free to write tests that help you develop your thought or solution. Not all of those should be checked in
2. Ask yourself: “What test do you need to write in order to protect this code from your future self next year when you have forgotten everything about this?”
People who don't test don't test because they don't know how. They are hacks.
Also hopefully interesting, the opposite of “don” is “doff”.
Or maybe they're just doing a different trade off. Do you unit test all of your shell scripts and jupyter notebooks?
If not, why is that? Is it that some code is either obvious enough or doesn't warrant having tests for it?
There's a spectrum of testing, from zero tests (which is fine for throwaway code), to testing a few key components and having end to end tests for the rest, to having 100% branch coverage. One should write enough tests that they're confident about the quality of the software that they're shipping, and no more, in my opinion.
* BE tests are easy to write, catch lots of important bugs, and generally speed up development.
* FE tests are painful to write, rarely catch meaningful issues, and don't really save any effort - devs still need to click through a UI to make it's correct.
* E2E tests were even more painful to write. Likely caught more than FE tests, but not enough to justify their even more insane overhead.
----
Typescript does wonders for FE test coverage. The bugs we do run into tend to be silly oversights (empty arrays, FE <> BE type mismatches, etc). Annoying, but quick fixes. Not worth a 30% overhead on every feature to save 5% occassionally.
I have nearly 100% unit test coverage for a couple of the base libraries I've created, but yeah, writing tests was slowing me down significantly for the rest.
Once I have something where chunks of the application aren't being scrapped every other week, I'll start adding unit tests to it too.
But really, I can see it. Firstly, a lot of engineers suck are writing tests. White box unit tests with interfaces and mocking everywhere tightly couple code blocks together. Integration tests are nice but take a lot of effort to get right while also keeping them performant. When (not if, remember we're talking about an early stage company) the codebase and/or the database schema changes dramatically it will often break the tests which is just another hangup.
In my pretty long experience it is a product issue, not tests'. And that is a kind of issue which sets the tone for that product development for years ahead.
>the codebase and/or the database schema changes dramatically it will often break the tests
That shouldn't break integration tests [much].
It also requires considerable skill to build and maintain, and not every team has that...
...in which case you might have a less good and useful test suite.
I think it is silly to use this one sentence without a context and start arguing. The whole paragraph in the article makes sense, gives context and advice is reasonable.
Something that's missing from this list that I find extremely useful:
An unstable environment with actual users doing real things with it. They could be internal, they could be advisors, they could be customers who are also investors or are willing to be testers for a discount.
That works really well if your code is designed to fail loudly (validations, assertions, etc...) and you have a culture of fixing issues immediately.
the latter is the only hope, but practically no one is willing to commit to it
now when I talk to people about making their product be something that development can be done on again - I try to figure out how serious they are. almost never.
It might be just me and a terrible string of bad luck bad I've seen so many projects that were kickstarted preemptively due to company politics or just management wanting to secure additional funding.
By the time (usually between 3 to 9 months) business stakeholders came to an agreement on what they actually wanted the team developed so much technical debt that it's virtually impossible to untangle from all bad decisions for the rest of the development cycle.
That's why I'm a huge advocate of starting simple, with a monolithic approach, to tread the water and have enough time to figure out the true objectives of the project first.
The primary goal is obviously to produce useful and targeted feedback as early as possible, but more strategically it elevates the status of QA, making it a gateway job to developer, SRE, or product management. In some organisations the QA function is perceived as a dogsbody role, and a reframing helps to flatten the pecking order and improve access to the talent pool.
Before we all went remote work, I'd always recommend adjacent seating for quality folks, to exploit the natural proximity bias. Alas, I'm still not sure what that looks like today.
On the FE, we noticed:
* Most of the bugs were in UX/UI that are very hard to capture in a test suite.
* CSS/UI bugs are better handled with a good UI library. We have an _amazing_ UI dev that's built a really stable UI library for the team.
* It took forever to write FE tests.
* It took even longer to run FE tests.
* Even well-written FE tests miss a lot of subtle issues.
* We rarely had "random" bugs. UI bugs were almost always in the code we changed. Easily detect with quick click testing.
What _does_ work is front end integration tests with cypress/selenium. You just write some basic tests that click through the flows and see you got to the end. This catches issues all the time where the api response or something has changed unexpectedly and the UI gets stuck half way on an error.
It’s also super simple, it’s 10 likes of looking up elements by ID and running .click(). They hardly ever break from normal UI changes and prove that the app is mostly working as intended.
Switching to TypeScript is also a big one. It’s almost like half a test system given to you for free.
I would also state that "10 likes of looking up elements by ID and running .click()" is a big part of why e2e tests are flaky and hard to maintain. Good e2e tests require good engineering.
We've found that Typescript and a bit of boilerplate has almost entirely eliminated these types of concerns for us.
In a similar manner, putting legacy code under test tends to remove a bunch of bugs; not because the tests find them, but because when I refactor the code to be testable, the bugs in the code become much more obvious.
Sigh, such a misguided quota.
This is a big one. It seems the industry (at least to my awareness) has almost totally rejected Gitflow for trunk based development in the last few years. It’s a beautiful idea in theory, but a complete headache in reality.
> Use feature flags
Be careful here. Adding feaure flags means you now effectively have N^N builds (where N is the number of flags) in production, each with their own possible quirks and integration issues. Keep them to an absolute minimum, and remove them as soon as possible.
Stacking is a wonderful workflow to accomplish this: https://graphite.dev/blog/post/N8SOs49y4bYdV1Vjf5qE
Also what happens if during the review of PR #1, the reviewer suggests something profound that would fundamentally change the direction of future commits?
All that stuff can come after you have proven your software makes money / users want to use it.
Just get the damn thing written in the quickest, most crappy way necessary for speed.
The code doesn't need to be organised well. It doesn't need to be documented.
The only thing that matters is to do your best to avoid security holes.
No tests. Stop testing, apart from manual runthroughs that the CEO can do in between trying to raise funding.
My eyes roll when I hear of startups creating "good processes" for development at any level, let alone CI/CD and tests.
Build the MVP so that it's good enough to start solving whatever it is you're solving and get your first customers.
If you can't get customers, what's the point of having a sophisticated codebase with tests, etc?
And in fact every minute spent on writing and running tests (and there's always ALOT) and CI/CD is wasted, wastes money and time, leading to longer development time until the software can be put in front of users.
Therefore CI/CD and tests and all their associated processes are a red flag and increases the likelihood a startup will fail to bring its software to market.
Developers who have always worked in well organised environments have really deep rooted trouble getting their head around this way of thinking. If you say to an experienced developer "no tests! no CI/CD!", it's likely they will immediately decide you are an unprofessional hack.
E.g.: it doesn’t matter what fancy SIEM tool you’re using if you never patch anything and the root password is “password”.
“But the vendor assured us that it would solve our security issues!” is the inevitable reply.
The sum total of time spent on CI/CD and tests is not the time it takes to get it started, it's the time spent executing the tests/CI/CD into the future, thinking about the tests/CI/CD, problem solving the tests/CI/CD.
All these things require ongoing time which takes time away from developing the application.
At my last startup, I wrote totally throwaway code on all our initial prototypes. Once we had something that looked like it would hit product-market fit, I hired a team, we threw away the shitty codebase and started fresh.
If you don't throw it all away after making the kind of mess you describe, then you are going into the company's most crucial time with a boat anchor around your ankle and the water level rising. I'm very glad we did, as starting fresh with tests, CI/CD, and other quality practices let us be extremely responsive to my cofounder's needs and the things we learned from every release.
This is why an experienced developer is sometimes the wrong person for the job to build a startups first version, unless they are able to fully embrace the mindset of prioritising speed over everything.
Beware of experienced developers who get the job of writing a startups first code and who start talking about tests/CI/CD/feature branches and all that stuff.
Avoid security holes and take backups, that's all that matters and anything else should be actively avoided.
As an example, I did a startup circa 2010. We raised money on an idea that VCs were very excited about. So much so that we found out later that other VCs had invested in similar notions. We spent a few months building disposable prototypes to test the value prop. It turned out that users hated the idea no matter what we tried. So we killed it after spending maybe $100k and did something else. Job done! But the other companies persevered, building full apps, and each burned millions before they realized there was no future in it.
So in that case, yes, you don't want to invest a lot. But a few ideas later we had something that looked to have real product market fit. At that point, we threw away the code and started fresh with solid tests, CD, and other quality investments. Why? Because the job we were trying to do had changed. It had gone from hypothesis testing to actually serving users and building a real product. If you want to get that job done well, you can't afford to have a garbage code base.
I like these companies. Share the names and I'll have my sales guys reach out.
...if only I had a cent for every time I've had this conversation with an engineering manager: Manager: "Just live with the mess for now. We'll fix it later when we have money." Later: "Can we fix the mess now?" Manager: "No. We're making too much money now. Blah, blah, blah. Opportunity cost."
That is ludicrously bad advice. Setting up a straightforward CI/CD pipeline can take a couple hours and it instantly provides huge productivity benefits.
I worked for a startup that didn't have an automated deploy pipeline and didn't have any tests. It was a complete shit show from day 1, and I will never work for that chaotic of an environment again.
I am certainly not for over-engineering early in a company's lifecycle, but there are simple, "table stakes" things like use of SCM, automated builds and CI/CD pipeline. If I don't see those I assume your engineering team doesn't know how to build software fast.
Kill CI/CD with fire for the sake of speed.
"Craptacular development mindset" is needed.
Sure, same as everything else that is overengineered. A simple, no-frills GitHub Action (or equivalent) should not take longer than a few hours to implement. If you need more time than that, it means that you have an overly complicated deployment process, which is a time sink when done manually anyway.
Your software should meet the minimums standards required by your target user.
That's the "viable" part of Minimum Viable Product.
In a similar vein, you cannot release a new antivirus product that does not detect viruses, or security product that does not make things more secure. Software development has (or should have) defined priorities - and if accurate financial transaction processing is a base requirement then that's the minimum viable product you need to deliver. And if it takes tests to do that, then that's what you need.
Being a startup doesn't mean your tech has to be garbage. So many miserable devs hoping that "after series A" they'll get to refactor that spaghetti code mess (they won't) and seeing their velocity slow to a crawl because they can't make a change without worrying the whole thing is gonna break.
2. you are wildly unclear about your preconditions. I am pretty sure you are talking about the initial stages developing an MVP for a startup. Other comments are probably referring to the startup growth period following the MVP.[1]
3. You might be right in some circumstances. But your assumption elsewhere in comments that CI causes over-engineering is incorrect: any team that has an over-engineering problem will generally be in trouble, which is not caused by CI.
4. You are replying to many comments dismissively, almost attacking with your “facts”. If your argument is strong, others will answer for you. “ Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.” — https://news.ycombinator.com/newsguidelines.html
[1] I’m using this definition for startup http://www.paulgraham.com/growth.html — personally I dislike PG’s definition of a startup (paraphrased: designed for fast growth). Should a business that is trying-but-failing-to-grow not be a startup? A business designed for slow growth but large market is not a startup? PG gives Google as an example of a startup, yet that contradicts his own essay: didn’t Google have a lot of luck, and it isn’t obvious to me that Google was designed for growth, more that the opportunity knocked and Google jumped on that opportunity, successfully. Some of PG’s definition feels tautological; 20/20 hindsight.
Edited: made politer.
Agreed. I have made an error there.
>> 4. You are replying to many comments dismissively, almost attacking with your “facts”. If your argument is strong, others will answer for you. “ Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.” — https://news.ycombinator.com/newsguidelines.html
Agreed. I have made an error there. I should choose a better tone. I do work hard to get the tone right but not successful today obviously. Apologies if this has offended anyone. I usually delete things that I feel are too negative, adversarial, preachy or otherwise not the right tone.
>> 2. you are wildly unclear about your preconditions. I am pretty sure you are talking about the initial stages developing an MVP for a startup. Other comments are probably referring to the startup growth period following the MVP.[1]
The comments I make are about a company pre product/market fit, doing the first build of its software before opening the doors to customers. I do agree that the startup period continues beyond this perhaps for a number of years, at which time development priorities might change.
>> [1] I’m using this definition for startup http://www.paulgraham.com/growth.html
For reference, in the link PG says "A startup is a company designed to grow fast.". This seems overly narrow because it seems self evident that there is lots of startups that have different objectives to fast growth.
My definition for a startup is more like Steve Blank's: “A startup is a temporary organization designed to search for a repeatable and scalable business model.”
I think you (a) have been burned (and I'm not denying testing can be a waste of time) and (b) not being a dev yourself you don't fully understand that this isn't a black and white issue or that you are actually paying a massive velocity cost by doing zero testing. Your POV is akin to advising people to not drink water on a hot, strenuous hike because "every time you lift your arm to drink, you are wasting energy that could go to your legs". Uh yeah, in both cases it's a little more complicated than that. Definitely don't waste an hour making a gourmet meal (and yes I get sometimes devs do this), but there are a spectrum of options and there is a sweet spot. How about "drink some gatorade and eat a power bar without slowing down"? Getting that balance right isn't trivial but it is possible and the payoff is huge. Which goes to the other point you made about not hiring expert devs, man, you've been hiring the wrong experts if you think fresh, unsupervised bootcampers are a better choice at any stage other than "can you make me a framer prototype over the weekend?".
Why do you think I'm not a developer?
Here is a list of OSS developer security tools: https://github.com/boxyhq/awesome-oss-devsec
I think better tooling around git will help a lot with speeding up development teams.
We also found feature flags slowed us down a lot in the early stages. I feel these should only be done for some features (e.g. an explicit AB test, a large feature with a lot of technical risk). Feature flags create a ton of complexity, and are often not worth it.
That essay was written based on 3 three pre Series A companies, which didn't have right processes in place and was struggling to ship fast enough. Weekly deploys is a first step. It's difficult to impossible to deploy more often if you don't have good tests coverage.
> But, on another side tests slows you down as you build features. When your team writes tests they need 2x more time to release any feature.
Tests are an investment in fast future development. Last I did a startup, we did not write tests for anything throwaway or experimental. But once we decided some chunk of code was a keeper (e.g., experiment had a positive result), we wrote tests.
With just a team of 6, we were releasing a few times a day, not just once a week. And we had no "QA freeze" nonsense, because our tests were solid enough that we didn't need manual QA to release.
That said, I hired for people who were good at writing tests, and we did pair programming with frequent pair rotation, which gave us a big quality boost and helped keep our test suite lean and effective. I can believe that some startup teams have low enough levels of testing skill that "they need 2x more time to release any feature".
I mean, they are, but what's the roi on that investment? Not all investments are good.
90% of start-ups fail, the chunk of code that's a keeper still has a 90% chance of being thrown away.
Before you have signs of product-market fit, I think people should write throwaway code and then throw it away. Once you have something worth investing in, I think it's time to invest. Having high-quality code in areas where you're confident it's valuable enables rapid experimentation in the areas where you're not sure yet.
Some reasonable advice in here otherwise.
You’re still waiting to release features that are done and turning on a whole week’s worth of features at one time, but it’s even worse because you spent a bunch of extra time building it to have feature flags.
The modern "feature flag" is more of a knob that can be slowly turned up, along with ideally a way to send shadow traffic through the feature even before any % of real traffic is allowed through.
In that proper implementation of feature-flags, the overall benefits (easy rollback, graduated rollout, selective rollout based on user characteristics, better ability to manage scale) outweigh the rather-trivial engineering cost of wrapping a few if-clauses around the feature.
> rather-trivial engineering cost
I personally disagree with this. To implement feature flags your engineers have to reason about all the places where the feature needs to be on/off, implement those flags, write and maintain test cases for both the on and off case, rollout the code, then go back and refactor everything to remove the flags afterwards.
It’s not insurmountable. But it’s not trivial in my experience.
I respectfully disagree, because I'd say that a "release" ought to be defined from the viewpoint of a given individual user. From that person's perspective, they are indeed seeing things change only weekly, and the details of how that's done (e.g., feature flags vs giant deployments) are mostly irrelevant to them.
> It’s not insurmountable. But it’s not trivial in my experience.
You are right. Honestly, I'm so used to having to sell the idea that I just reflexively wrote it that way.
Thank you for listing out the different pieces. You're absolutely right there is a lot of extra "essential complexity" that gets added in. Even if the "actual" coding work is "just a few" extra if-clauses and tests (which is how I'd sell it ;), everyone has to hold a lot more in their heads.
Personally, though, I would still insist that it's way better than the essential complexity that gets added everywhere by coupling "releases" with "deployments." In my experience, once an org lets product-release-tempo dictate engineering-velocity-tempo, the bureaucrats essentially have a chokehold on all engineering decisions.
The organization slowly grinds to a halt; high-value customers start getting picked off by nimble, faster-moving startups; and the circle of life continues.
Feature flags are IMHO a key technique for a startup-that's-getting-bigger to hold off this particular mechanism of decline. Non-technical stakeholders at all levels can be easily waved off with a simple remark of, "Well, we'll feature-flag it." And engineers can stay much more empowered, even as the company grows much bigger.
Both Google and Facebook use monorepos
If you're not successful, then you won't have any scaling problems.
These are at odds with each other. In my previous role at a startup we deployed on average twice per day per developer. The automation necessary to make this hands off and reliable wasn’t actually that much. It rarely caused issues.
I write apps (which may include some fairly significant backend stuff).
Some of these suggestions apply, but not all.
I ship weirdly fast, anyway,
I think the pandemic broke us.
I very much agree with break features into many small pull requests. We merge constantly even (especially) if the code path won't be hit yet. Just get it into production. Makes the code review faster, the complexity lower, and lets you focus on one thing at a time.
I think no tests makes you slower. We paid a bit of cost to start writing tests, but now we can deploy 10 times a day because we know the mission critical stuff is working. It's more about testing the right stuff. Decide what can break, and leave it untested. What can't break should be covered by tests. We do stuff with money, all that is robustly tested. The UI isn't.
Code freezing / weekly releasing is counter to CI/CD. Just make your deployment take less than 10 minutes, and ship code constantly.
Use feature flags to decouple Deployment from Release and you’re fine.
But management sees the capability and says add a flag for customer Y that changes how this page works.
And after it’s enabled for everybody, remove the flag.
This is great until it's not. We're starting to hit this tipping point on our code-base.
The challenge for us is it'd likely take an engineer a week or two to optimize our test suites. That's enough time for them to ship meaningful features.
Instead, we do occasional freezes for important releases. Not sustainable, but it's a good enough bridge.
I need dev time to make the builds understand their inputs, and then cache more aggressively than we are, or something. (But I lack the dev time.)
caching and promoting images is an absolute must. caching images has it's downsides, too though. it can be a longer time period between when something in the dockerfile breaks and when it's noticed.
generally speaking, my pipeline is generally 10-40% build time, 50-60% integration & system test time, 10% deployment time
The system tests that're run deploy the entire dependency graph of services that my project depends on, which can take a bit to settle, has a high chance of transient failures relative to other parts of the pipeline, and notably has a fixed test run time.
That last bit really raises the lower bound for how long the pipeline as a whole takes -- number of runners doesn't matter.
The pipeline generally takes between 25-45 minutes, and runs on each push and is triggered by changes to dependencies to ensure backwards compatibility.
I wouldn't do anything to make it take longer than it does now, and it can be frustrating. but the pipeline will catch things that would otherwise slip through the cracks.
E.g. we went with that route with GH Actions, until our tests didn't fit into the memory of the provided machine sizes anymore (shouldn't be a problem anymore as GH provides options for bigger runners now). So we had to spend 1-2 days on setting up a self-hosted runner in a way that it wouldn't be too heavy in maintenance, and then probably another 1-2 days of rework over the next few weeks to iron out the kinks.
The occasional freezes mean future tests written do not need to be optimised (there will be a freeze for important releases anyway), so the unoptimised-tests problem will grow worse the longer you use this strategy, and the cost to rectify it will grow, until your release frequency collapses on itself and everyone complains that "there's no solution to this problem".
These are the sorts of things I think it's very important to nip in the bud. Spend the two weeks now, when it only takes two weeks! Much better than putting everything on hold for six months to fix it four years from now when it becomes untenable.
That does seem like a possibly worthwhile choice though? A "stop chopping the tree to sharpen the axe" moment?
But, maybe spend a day on it. I've been to places where the build took hours and there were obvious 30min solutions to fix it (which I did, in some cases).
Consider it an investment. 2 weeks now gives you how many more features over the next six months due to increased dev velocity?
Bad tests can absolutely make you slower. Good tests can be existential.
The benefit is that the tests highlight what changed in the app. So when you update a library and 1+1 changes, it’s an alarm bell to investigate further because you didn’t expect this to happen.
Test_OnePlusOneEqualsTwo()
is not as useful as
Test_CanAddTwoNumbersTogether()
Test for behaviour, not implementation.
At an early stage startup with a small app it might not be as big a problem, but where I'm at right now the app is over 10 years old and our test suite with many many parallel workers is still well over an hour cycle time to get that sanity feedback on a commit, and it's painful.
We manage daily, but even that is a miracle. If you're building enterprise CRUD app #310979707 or SaaS project #2009879, sure, deploy constantly. But a very significant number of projects simply cannot do that.
I see that I had assumed we were talking about web apps. This assumption narrowed my perspective, and was an error in thinking on my part.
(I'm not trying to imply that that's a mistake you've made; it's just something that it's easy to forget as a manager of programmers, and is maybe worth pointing out explicitly; you need a way to measure the team's productivity, but it's very hard to find a good, measurable proxy for 'productivity' which isn't eventually going to end up being gamed by the whole team!)
I would not judge an individual developer's productivity on commits per day, agree that would be foolish.
1. Pre Series A companies, which didn't have strong engineering team from start and now struggle to ship. 2. Idea/Pre Seed companies, which just want to ship fast and raise funds. That kind of founder usually have 'friends & fools' budget.
1. First kind of projects usually don't have enough tests coverage to deploy safely more often. Typically I would set a goal to deploy twice a week in 6-12 months and build a plan of how to achieve that. 2. Usually I advice to not write lots of tests for the first ~6 months to focus on shipping faster. In my experience products often change dramatically during this period. But a) there are exceptions. Some domains (e.x. finance) are critical to write more tests. b) tests infrastructure are always set as part of CI/CD process, so engineers can decide and write them for area of high importance.
This is the important point.
Anyone talking about code coverage percentages is already making a mistake. percentage of coverage isn't nearly as important as WHAT is covered, and it's not a good proxy for it either.
Ideally the client won't let them occur. But you should still verify the server doesn't give up the goods.
I guess you could "manually" test by using the command line? Sounds like a lot of work. In my experience, more work than automating it.
At my current job (fast-growing mid-stage startup), any developer can push when they want, but it works a bit differently. Once the release is staged, everyone who has a commit on staging is responsible for verifying their feature. Then the developer goes through a "release checklist," which involves finding a buddy and jointly running through a list of manual tests to validate key features. After that, they can push. They probably push a few times a week.
I feel like code freeze/manual QA/weekly push is something of an anti-pattern. Developers are disconnected from the process of releasing their code, and take less ownership. While in practice manual QA should lead to finding more bugs, in practice it can lead to more defects.
saves 48 hours on releases