Launch HN: GrowthBook (YC W22) – Open-source feature flagging and A/B testing (growthbook.io)
Developers often launch a feature without understanding the impact it has on their users and business. This is a big deal, because only 1/3 of product launches actually improve the desired metrics. Of the rest, 1/3 have no effect, and the last 1/3 actually hurt [1]. The best way to measure this is to use feature flags and controlled experiments (A/B tests).
Jeremy and I worked together for 10 years at an ed-tech startup as CTO and software architect. We spent far too long just building and launching features without really knowing how they impacted our users and if they were adding value to the company. We had product analytics, but there was too much noise in the data to draw real conclusions. We knew the “right” way to do this was to build feature flags and run controlled experiments, but that was daunting for our small team.
We looked into 3rd party tools, but it bothered us that they didn't use our existing data warehouse and metric definitions, and we really didn't like the idea of adding an API call in the critical rendering path of our application. We also didn’t want to send our data to 3rd parties, didn’t feel good about vendor lock-in, plus the vendors were expensive. So, we did what any engineers would do—build it ourselves. After all, how hard could it be?
After a couple painful years, we hacked something together that (mostly) worked and used it to help grow revenue 10x. We started talking to other teams and realized just how many larger companies spend years building these feature flagging and experimentation platforms in-house because, like us, they couldn’t find any tools that met their needs. So we took everything we learned and built the tool we wish had existed back when we started.
GrowthBook is an open source platform for feature flagging and A/B experimentation. Our SDKs are built to be fast and cache-friendly. We take data privacy seriously and don’t use cookies or deal with PII. We sit on top of your company’s existing data warehouse and metrics so you can maintain a single source of truth. We’re open source (MIT), so you can either self-host the platform (with Docker containers), or use our managed cloud offering.
In GrowthBook, feature flags are added and controlled within the UI. Engineers or PMs can add targeting rules (e.g.”beta users get feature X, everyone else does not”), do gradual rollouts, and run A/B tests on the features. The current state of features are stored in a JSON file and exposed via an API or kept in-sync with a cache/database using webhooks. Engineers install our SDK and pass in the JSON file. Then they can do feature checks throughout their code (e.g. `if feature.on { ... } else { ... }`).
For A/B test analysis, a data analyst or engineer connects GrowthBook to their data warehouse, then they write a few SQL queries that tell GrowthBook how to query their experiment assignment and metric conversion data. After that initial setup, GrowthBook is able to pull experiment results, run it through our Bayesian stats engine, and display results. Users can slice and dice the data by custom dimensions and/or export results to a Jupyter notebook for further analysis.
We’re used by over 60 companies in production. We have self-hosted and cloud versions (see our pricing here: Is there a way to self host without the neatly packaged docker version? Reading what you are running implied it could work on my shared hosting playground that doesn't support docker, but all else (with probably a good bit of tinkering from my side) could run smoothly. Maybe I could - if I get it running - write a lab tutorial for hosting on uberspace (German hosting of yet-download fame). As said: congrats and great site. A lab tutorial for uberspace sounds awesome! We're happy to help you get things set up if you run into any issues. There are _so many_ types of services that were similarly disrupted by solid open-source implementations due to their price: business intelligence (Metabase), live chat widgets (Chatwoot), monitoring/observability (Grafana). This feature set is even more crucial than the ones above so I'm surprised a default open-source implementation hasn't emerged. Excited to see how this progresses! It was upgraded internally to Sevenpack but something like GrowthBook looks much nicer! Re: Optimizely - Optimizely is a walled garden, you need to push all of your metrics as events to Optimizely from your apps for any experiment analyses. Eppo and GrowthBook both connect to your data warehouse and support any metrics you can design in query form using your existing warehouse. The other missing part in most feature flagging tools is the analysis side. Most tools let you run an A/B test, but it's up to you to track the data, process it, and run it through a stats engine. We built that part directly into GrowthBook. 1. https://matomo.org/
2. https://github.com/growthbook/growthbook/issues/86 A lot of big companies need to gradually introduce a new feature to an audience while also seeing how that feature impacts their metrics. If they go with just feature flagging, they have to build their own logging and analysis. But if they go with experiments, then they're overly constrained by that analysis since experiments are meant to be statistically rigorous. It's a tricky problem which I think statsig solved with "pulse metrics" and "feature gates". I think this isn't just marketing; I think this reflects a deeper understanding that you can do A/B testing on top of a feature flag and not consider it an experiment. It's an innovation that reduces inter-organizational strife on the decision to launch a feature to all users, i.e., ship. If everything with automated metric analysis is considered an experiment, then the shipping decision only be made after analysis is performed. That leads to tension when Product Managers have already decided to ship and will only rollback if something hurts their metric. Data Scientists will then question why their "experiment" is only running for a day, and then question the decision to ship entirely. You can imagine how many meetings a misdefinition spawns. The sooner you address these definitions, I think the happier you'll find your enterprise users. Makes a lot of sense with an experimentation tool that just works on top of your existing customer data. This is the way to go for data tools in my opinion. Makes it so much easier to adopt. What we have done, is just exposing views on top of our regular click-stream data. With this approach, we can focus on having one data source of high quality, in stead of scattering data around to multiple third party tools. And we don't have to deal with lot's of data processing agreements and so on. I hope more and more data tools moves in this direction. Sending data to a third party is starting to become old school :) We're a small startup moving incredibly quick. I was shocked at how much setup time it took to get a simple feature flag into our app with other services like Darkly. We had Growthbook working in roughly 15 minutes with under 10 lines of code changed. Feels a lot like the "a-ha!" moment I had with Plausible.io. This is all we will need for a long, long time. - why is the submission text lighter than comments? - rgb(130,130,130) on #F6F6EF is a bit hard to read - rgb(130,130,130) is or is too similar to downvoted text color - rgb(130,130,130) is the same color as visited links, so it makes links users visited not appear as links, like the link in "We did a Show HN 6 months ago (https://news.ycombinator.com/item?id=28088882, ..." to me Back in the day if you made open source software SaaS wasn't usually an option, but what you did was offer to install it on premise and offer support for a truckload of money. Often enterprises liked this because they wanted someone to call/blame when something went wrong. I wonder if these new companies could still add that to their offering. Free open source, SaaS if you don't want to host it, or we'll install it for you on your premises and offer support for piles of cast (This isn't meant for you guys specifically since I see that you basically offer this under "Enterprise") BoldVoice has been using GrowthBook for about ~6 weeks or so now, and it was super lucky that we found these guys right as we were considering way more expensive options like Optimizely (...6 figures). The tooling is pretty intuitive and Graham and Jeremy have been providing stellar support. There's still a ton to build here and I'm excited to see what these guys add to the platform. And why MIT license not AGPL? Personally, I am a happy user of PostHog (gitHub.com/posthog/posthog) which also supports experimentation and feature flags alongside product analytics. Isn't this a bad practice in general? Not only you split your (potentially limited) sample count, but it is very unlikely that there is ZERO interaction between A/B tests. Even if you test something on one page and another thing on a different page, there might be some interaction between the two changes. There are two ways to deal with the bigger interaction effects. First is to predict which experiments will meaningfully interact ahead of time and split the samples between them (making the tests take longer). Second is to run tests in parallel on all users and look at the data to determine interaction effects after the fact (and potentially need to invalidate some results). In our experience, a mix of these approaches works best. It's really hard to predict meaningful interaction effects ahead of time, so save that for the really obvious cases (e.g. black text on a black background). For everything else, the benefit of running more experiments usually outweighs the cost of occasionally needing to throw out results because of interaction effects. It's much better to run 10 tests and need to throw out 1 than it is to run 5 tests.46 comments
[ 3.8 ms ] story [ 117 ms ] thread