Not to divert from the subject of this post, but could anyone explain to me what a feature flag looks like? How do you go about implementing one? There was a post here about trunk-based development which mentioned feature flags quite a bit. They seem like they are quite useful but I've yet to encounter one.
They can be all kinds of things, at its most basic, its just a way to put features behind some kind of toggle, that are only shipped or enabled if something is determined to be toggled as "on".
For example, you can use `#ifdef` in some languages like C#, C++ and C to provide features based on platforms and/or language versions, so builds only contain code when those statements are true. With things like esbuild, webpack, or vite, you can use their `define` options or in the case of webpack, something like the `DefinePlugin` or `EnvironmentPlugin`, to turn `process.env (like `process.env.NODE_ENV` etc) or other synthetic globals to certain values, which enable features (and ideally, tree shake out unused ones for the build). These are typically done and used at build time.
These can also be done at runtime, by either hardcoding a feature behind some kind of statement to evaluate its usage, or using more sophisticated feature flagging systems, such as:
- Split IO[0]
- Launch Darkly[1]
which let you toggle the feature flags dynamically during runtime. This is how you can do things like A/B testing, for instance, or allow only certain users access to a feature.
These are the most common examples of feature flagging I can think of that I think most have seen, even when its not explicitly noted as such.
I'm not entirely sure that gating vs flagging should be defined differently
Note: I have zero affiliation with either of these products I mentioned, I've just used them before
> I'm not entirely sure that gating vs flagging should be defined differently
I think it’s interesting to contemplate. I think they’ve both stuck around at Stripe for so long because there’s something useful communicated by the (correct) choice, and having different implementations with different trade offs based on the use case.
OTOH, every single user of the system has to understand the differences in order to consume them correctly, and even the search/browse is (was?) separate, iirc. I suspect a big improvement would just be to unify the web administrative interface: clearly indicate which is which, but allow common operations without needing to know ahead of time whether something is a flag or a gate.
A flag function that decides a boolean decision could have a signature such as f: context_t -> bool where context_t is a value type containing fields such as: uuid of the current user, uuid of the node or region we are deployed to, the current time, the current version of the application code.
Feature flag platforms such as launch darkly can be thought of as a way to manage and deploy new implementations of decision functions into running application processes. E.g. dream up a class of decision functions with the signature f: context_t -> bool that could be serialised and sent over the wire between two machines.
For large muddled IT organisations, toggling a flag in a production application by having a developer hardcode it or toggling a flag by changing static application config and redeploying might require 15 minutes of development time followed by days of integration testing and meetings and change approval lead times. In these settings, having some kind of feature flag platform deployed that allows application behaviour to be toggled very rapidly without coupling it to some kind of IT software deployment might let the organisation react and run experiments two or three orders of magnitude faster. Give product managers or "the business" the ability to adjust application configuration live in production with the feature flags without having layers of slow process in the way.
If your application contains a large subsystem where some decision or logic is evaluated by a pure function, it might be a fun exercise to think through "what if we made that it possible to swap in new implementations of that function into a running process in production, like a feature flag". E.g. if your subsystem can be described as an one big expression tree, and there's a way to serialise that expression and send it over the wire, and have an interpreter in your application process execute it.
Maybe the natural and logical conclusion to this is that every production application should be Joe Armstrong's universal server:
> This was a problem especially for newer engineers, who would get (unsurprisingly) confused as to whether they should be using one or the other, and even senior engineers would regularly forget that the flag they’re looking for is actually a gate (happened to me five times a week). Ideally, these would be just be one concept
That's what you get when you try to be clever with your vocabulary rather than using longer and less ambiguous terms
if feature_enabled("new neat thing") {
doNewNeatThing()
}
In principle that's it. How exactly it decides true vs false can be as complex as needed - hardcoded, config files, fully dynamic, etc. It's defined more by the use than by what it is / how it works.
E.g. a number of teams I've seen use them to dogfood their team's features (e.g. via user ID) for a brief period as a final check before wider release (which is also rolled out by that feature flag in %-batches or other criteria). It also gives you an easy dynamic "off" switch if something isn't working out or is causing errors.
Server-side that may or may not be particularly useful, but in apps with week-long review delays it can be the difference between "a brief burst of issues for a few people" and "our app is totally broken for a week, customers are ANGRY".
There's a good blog post from ThoughtWorks (MartinFowler.com) about feature flags, the different categories, and code examples[0]. The post explores different feature flag implementations and tradeoffs, depending on the longevity and dynamism of the feature flag.
I am using home grown feature flags, so in a way I know what they are. Where I sometimes feel like I’m doing something wrong is when they are mentioned as some silver bullet for complex deployment scenarios or real safety for testing in prod.
In my mind they only really work for a subset of features that are relatively clean: show this tab in the app y/n, mount this react component y/n. But the most complex and dangerous features really aren‘t like that. They involve sweeping changes that introduce incompatibility with previous behaviours, schema changes etc.
I don‘t see how you can flag that stuff without littering conditionals and making the code unpredictable in the process.
There are many impressive looking feature flagging platforms, but as far as I can tell, they all stop at handing you a boolean. Feels like serving booleans to users is the easy part?
I've been looking into using Unleash (https://www.getunleash.io/) for this sort of thing. Seems like that would be a better option than rolling-your-own, particularly if you have the infrastructure for it. I haven't actually gotten too deep into it to know what limitations might come with using it. Does anyone have any experience with it?
Not totally sure Flipper is in the same category - that's more a Ruby-specific library, no? I know they have a cloud offering, but that isn't open source.
One idea to avoid keeping all the flag tokens in memory is to populate them dynamically based on information known at request time. For instance, if the request is scoped to a user and application, we can fetch the flag tokens related to that user and application in request handler middleware. Slap an LRU cache on it, and the performance is pretty good.
This doesn't work as well if feature flags need to target based on tokens that aren't known at request time, since you can't always asynchronously fetch additional flag tokens from a synchronous context. But, you can provide a separate async API for these situations.
Can anyone else comment on the SaaS companies like LaunchDarkly - where (afaik) the meta data for your feature flags are stored on some server external to your company and this "enables you to instantly turn on or off features".
I work in a big bank so this would just be laughed at, but has anyone used this? Why is it attractive at all? Surely there needs to be some release process of some sort ? what if they go down ? what if you stop paying them? How can you do useful things like "launch to only the Japanese users" without handing launchdarkly a list of japanese users?
It's one of the weirder SaaS areas - almost all the cost of Flags is in the writing of the code (and removing it later). Not the bit flip.
And if, as someone implies uothread, it's useful to speed up releases well, you are solving the wrong problem in very much the wrong way ..
It's much more than a bit flip - that's just the input. The output side is where tooling like this becomes useful, i.e., getting analytics on each variant. That's what lets you run A-B tests to optimise conversion, ensure a new feature isn't going to cause stabilty problems, etc.
And then there's the need for interfaces to allow changes on a live system, and giving certain abilities to non-technical users who want to change settings on a simple web dashboard.
If an external service goes down, the system should be able to degrade gracefully. I haven't used an external service, but I'd hope the SDK allows for a simple fallback strategy for each flag, such as random assignment or fixing to a particular value. Data collection should still resume once the external service is back up.
We've used ConfigCat (highly recommended). They publish a configuration file and the web client determines if the user is enabled or not.
Yes, this means we are paying for a UI and an SDK (plus an audit log and some integration into Slack and analytics) but it's still better than spending time writing a system ourselves.
You have to assume anything shipped is going to be public, and use sane defaults in code in case the service goes down. Once you move to that model, the idea of releasing code and features simultaneously seems reckless.
I've run into the feature flag scaling problem before - where every front-end server needs access to the current state of the flags, which can be a pretty large amount of state to pass around if you support per-ID flags for hundreds of thousands or millions of users.
In the past I've solved this with a per-instance cache, but that's frustrating because it means that feature flag changes now take up to 5 minutes to roll out.
Next time I need to build this, I think I'm going to use SQLite for it.
A SQLite database on each front-end instance could model arbitrarily complex feature flag rules, and even with millions of potential turned-on-for-user-X records would still only be in the order tens of MBs on disk.
Checking flags would be a fopen() call and and indexed SQLite query, with no network overhead, so practically free.
The challenge would be synchronizing down the central state to these hundreds of separate SQLite databases. That doesn't feel insurmountable to me - just having a process that polls a central source every 10 seconds or so for changes to apply since their last update could probably work fine here.
Don't do that. That sounds more like user preferences, which feature flags are not meant to solve.
Feature flags are supposed to enable features for some x% of users, which you can do by something like `user_id % 100 < x` (no database query required).
19 comments
[ 2.5 ms ] story [ 60.6 ms ] threadFor example, you can use `#ifdef` in some languages like C#, C++ and C to provide features based on platforms and/or language versions, so builds only contain code when those statements are true. With things like esbuild, webpack, or vite, you can use their `define` options or in the case of webpack, something like the `DefinePlugin` or `EnvironmentPlugin`, to turn `process.env (like `process.env.NODE_ENV` etc) or other synthetic globals to certain values, which enable features (and ideally, tree shake out unused ones for the build). These are typically done and used at build time.
These can also be done at runtime, by either hardcoding a feature behind some kind of statement to evaluate its usage, or using more sophisticated feature flagging systems, such as:
- Split IO[0]
- Launch Darkly[1]
which let you toggle the feature flags dynamically during runtime. This is how you can do things like A/B testing, for instance, or allow only certain users access to a feature.
These are the most common examples of feature flagging I can think of that I think most have seen, even when its not explicitly noted as such.
I'm not entirely sure that gating vs flagging should be defined differently
Note: I have zero affiliation with either of these products I mentioned, I've just used them before
[0]: https://www.split.io/
[1]:https://launchdarkly.com/
I think it’s interesting to contemplate. I think they’ve both stuck around at Stripe for so long because there’s something useful communicated by the (correct) choice, and having different implementations with different trade offs based on the use case.
OTOH, every single user of the system has to understand the differences in order to consume them correctly, and even the search/browse is (was?) separate, iirc. I suspect a big improvement would just be to unify the web administrative interface: clearly indicate which is which, but allow common operations without needing to know ahead of time whether something is a flag or a gate.
Feature flag platforms such as launch darkly can be thought of as a way to manage and deploy new implementations of decision functions into running application processes. E.g. dream up a class of decision functions with the signature f: context_t -> bool that could be serialised and sent over the wire between two machines.
For large muddled IT organisations, toggling a flag in a production application by having a developer hardcode it or toggling a flag by changing static application config and redeploying might require 15 minutes of development time followed by days of integration testing and meetings and change approval lead times. In these settings, having some kind of feature flag platform deployed that allows application behaviour to be toggled very rapidly without coupling it to some kind of IT software deployment might let the organisation react and run experiments two or three orders of magnitude faster. Give product managers or "the business" the ability to adjust application configuration live in production with the feature flags without having layers of slow process in the way.
If your application contains a large subsystem where some decision or logic is evaluated by a pure function, it might be a fun exercise to think through "what if we made that it possible to swap in new implementations of that function into a running process in production, like a feature flag". E.g. if your subsystem can be described as an one big expression tree, and there's a way to serialise that expression and send it over the wire, and have an interpreter in your application process execute it.
Maybe the natural and logical conclusion to this is that every production application should be Joe Armstrong's universal server:
https://joearms.github.io/published/2013-11-21-My-favorite-e...
I think that's the point of the article.
"feature flags" = "gates" = "circuit breakers" = "knobs"? really?
> This was a problem especially for newer engineers, who would get (unsurprisingly) confused as to whether they should be using one or the other, and even senior engineers would regularly forget that the flag they’re looking for is actually a gate (happened to me five times a week). Ideally, these would be just be one concept
That's what you get when you try to be clever with your vocabulary rather than using longer and less ambiguous terms
E.g. a number of teams I've seen use them to dogfood their team's features (e.g. via user ID) for a brief period as a final check before wider release (which is also rolled out by that feature flag in %-batches or other criteria). It also gives you an easy dynamic "off" switch if something isn't working out or is causing errors.
Server-side that may or may not be particularly useful, but in apps with week-long review delays it can be the difference between "a brief burst of issues for a few people" and "our app is totally broken for a week, customers are ANGRY".
[0]: https://martinfowler.com/articles/feature-toggles.html
In my mind they only really work for a subset of features that are relatively clean: show this tab in the app y/n, mount this react component y/n. But the most complex and dangerous features really aren‘t like that. They involve sweeping changes that introduce incompatibility with previous behaviours, schema changes etc. I don‘t see how you can flag that stuff without littering conditionals and making the code unpredictable in the process.
There are many impressive looking feature flagging platforms, but as far as I can tell, they all stop at handing you a boolean. Feels like serving booleans to users is the easy part?
For completeness, the other OS projects I know of are:
- https://github.com/markphelps/flipt
- https://github.com/featurehub-io/featurehub
- https://github.com/jnunemaker/flipper
Not totally sure Flipper is in the same category - that's more a Ruby-specific library, no? I know they have a cloud offering, but that isn't open source.
One more I found: https://github.com/checkr/flagr
This doesn't work as well if feature flags need to target based on tokens that aren't known at request time, since you can't always asynchronously fetch additional flag tokens from a synchronous context. But, you can provide a separate async API for these situations.
I work in a big bank so this would just be laughed at, but has anyone used this? Why is it attractive at all? Surely there needs to be some release process of some sort ? what if they go down ? what if you stop paying them? How can you do useful things like "launch to only the Japanese users" without handing launchdarkly a list of japanese users?
It's one of the weirder SaaS areas - almost all the cost of Flags is in the writing of the code (and removing it later). Not the bit flip.
And if, as someone implies uothread, it's useful to speed up releases well, you are solving the wrong problem in very much the wrong way ..
And then there's the need for interfaces to allow changes on a live system, and giving certain abilities to non-technical users who want to change settings on a simple web dashboard.
If an external service goes down, the system should be able to degrade gracefully. I haven't used an external service, but I'd hope the SDK allows for a simple fallback strategy for each flag, such as random assignment or fixing to a particular value. Data collection should still resume once the external service is back up.
Yes, this means we are paying for a UI and an SDK (plus an audit log and some integration into Slack and analytics) but it's still better than spending time writing a system ourselves.
You have to assume anything shipped is going to be public, and use sane defaults in code in case the service goes down. Once you move to that model, the idea of releasing code and features simultaneously seems reckless.
In the past I've solved this with a per-instance cache, but that's frustrating because it means that feature flag changes now take up to 5 minutes to roll out.
Next time I need to build this, I think I'm going to use SQLite for it.
A SQLite database on each front-end instance could model arbitrarily complex feature flag rules, and even with millions of potential turned-on-for-user-X records would still only be in the order tens of MBs on disk.
Checking flags would be a fopen() call and and indexed SQLite query, with no network overhead, so practically free.
The challenge would be synchronizing down the central state to these hundreds of separate SQLite databases. That doesn't feel insurmountable to me - just having a process that polls a central source every 10 seconds or so for changes to apply since their last update could probably work fine here.
Don't do that. That sounds more like user preferences, which feature flags are not meant to solve.
Feature flags are supposed to enable features for some x% of users, which you can do by something like `user_id % 100 < x` (no database query required).