HN Explorer

103 comments

[ 4.3 ms ] story [ 143 ms ] thread
hiatus
I like the idea of the flags being managed via normal change-management processes. In your experience, is it primarily developers in charge of enabling/disabling flags or have you seen responsibility diffused across departments (marketing, sales, etc)?
drichelson
I have mostly seen developers in charge of changing flags, but there is also great value in empowering other roles to change flags (One example: sales people can unlock features as they sell them).

Thankfully GitHub has an easy web-based Pull Request process. It's not as simple as a custom UI, but could be used by non-engineers to change flags.

vosper
It’s a trusting and non-siloed org that gives the sales team access to the repos
candiddevmike
> sales people can unlock features as they sell them

That's a licensing thing, not a feature flag.

yjftsjthsd-h
What's the difference? Either way you're turning on and off code that's already in the application. The only difference I can immediately see is that some people do feature flags as being on or off for the entire application instance, but I'm pretty sure you can do A/B testing with them at arbitrarily fine granularity and still count as a feature flag.
yoz
Not always. The most common counter example is beta features which customers opt into, controlled by the product team. It's too early for that to be part of the license, especially because pricing often hasn't been decided yet.

But there's also a surprisingly common situation with big customers who want to limit and control certain UI updates so they can ensure that employees are trained appropriately.

shagie
There are many different types of toggles of functionality. Not all of them are enabling code features. Well, they are but the why they are is what is at issue.

But it's all the same code under the covers. The functionality of what is enabled encoded into a license key - that's the same type of functionality as "does this deployment allow people to sign up by email" or "has this deployment enabled the super secret power user test functionality?"

But that's all under "if dataSource.toggle then {something}"

https://martinfowler.com/articles/feature-toggles.html

willsmith72
That's not a feature flag, that's a feature of the product. I would keep the 2 very distinct.
mikejulietbravo
I'm kind of surprised they haven't already built this into the LD core
snapcaster
If it's something controlled by the PR why is it even a feature flag? Seems like you lose a huge chunk of the benefit and might as well just change the code at that point. This to me seems like the wrong place to be controlling them, i certainly don't want to have to make a PR and merge it when a site issue is happening.

Open to missing something though, curious what others experience has been

alexchamberlain
I think the idea is that it's a separate flags repo?
snapcaster
Can you say more? I think i'm missing the point, in my mind that would make it even worse because you have all the problems I mentioned + merge conflicts being a possibility
Pet_Ant
Well I don't think you are doing active development on your flags repo. I think it's just using Git as a database for the what are your current feature flags.
irjustin
One repo is actual flags control-like-database.

So instead of controlling flags from a website, you get the benefits of git merge, PR's, reviews, documentation etc without having to rebuild it.

I like the concept since it brings accountability. But it's just a need that larger orgs have, but by that point have likely internally built a flag system and so transitioning is difficult.

Osyris
We do this at my company and another huge benefit is the ability to test config at a particular point in time. You can even git bisect to find which flag flip caused a regression.
drichelson
Correct: the flags live in a separate repo.
hinkley
That’s really an 80/20 situation. The flag repo fixes several large problems around visibility, coordination and state tracking of flag status, but introduces more friction into the system.

I’m not sure I have a comprehensive solution. I just know which ones I hate more than which others. The repo is the least obnoxious of the options.

drichelson
The PR is optional, but governance often requires documented peer approval for all production changes.

Flag changes can be pushed directly to the main branch with the correct repo permissions. When using the GitHub UI this involves just a little bit of typing and a few clicks.

>might as well just change the code at that point

If changing the code, running tests, building, and deploying is quicker and less risky then yes that makes more sense.

wewtyflakes
This seems like a reach. A nice benefit of having these server-owned configuration flags with a slick UI (like launchdarkly) is that they can be modified by people on the fly, and by people who may not even be engineers (like product managers). I imagine, that if the ask is that they instead get GitHub permissions, make a PR, wait for a review, etc, then perhaps you are not competing with launchdarkly. Though, having Git controlled server-owned configuration is still nice regardless.
willsmith72
Depends on the feature. In 99% of cases, I'd prefer an engineer to launch it and have the change tracked by source control
randerson
The sole reason I like feature flags is that I can quickly toggle off a change if it causes a problem. I'd hate to need to find someone to sign off on restoring service. Anything more than 1 or 2 clicks is just adding precious seconds to an outage. I've worked at places that gave broad implicit approval to developers to toggle away as needed. It worked well.
teeray
> I'd hate to need to find someone to sign off on restoring service.

In these shops, this gets handled via paging on-call engineers. The on-call is sometimes given more latitude if their actions are auditable.

mosselman
Imagine being called out of bed to turn off a feature flag.

This is nonsense.

__MatrixMan__
If not a PR review, how would you propose people should build consensus around:

> I'm about to do this thing to mitigate the issue, does it look like the right thing?

It doesn't need to be a code change, can just be a flags change, but if it's a change at all then why not pin it to a commit so that rolling it back is easy and so that the commit sha can be an indicator of which flags are where.

ryanmcbride
So you just flip feature flags in prod all willy nilly? No process?
kkyr
You're missing the parent's point. It's not about lacking a change control process, but about having the ability to instantly change the state of a flag when necessary. Moreover, these two can coexist effectively.
hinkley
With Continuous Delivery you should be able to roll back the deployment. If the last change is the only one you need to revert. Of course that stops working when you share the flag system across teams with separate milestones. Your guys are gonna want to flip toggles while my guys are thinking about flipping others.
wfme
There are more options than store in git, or flip willy nilly.

For systems of sufficient scale, it's fairly standard to keep flag changes outside of git so that they can be flipped without a pr. That way the flag change UI can apply other validation steps before any change is attempted such as ensuring valid enrolment ranges (no accidental overlap, and no accidental rollouts to 100% instead of 10%), and the associated rollout analytics can be shown alongside the changes.

You can also override things in emergencies more easily, which is the parent's point.

hinkley
But it’s also common to stick those flags back under some sort of audit system so we know why and how weird states got set. The simplest way is a separate repo with simpler rules for pushing to master.

Though you could also create your own audit system (just make sure it functions even when the entire site is down)

bottd
This is all decided by business needs, not by engineer preference. If devs aren’t pulling the levers it is good to expose them in an accessible interface, not plaintext.
hinkley
Let’s not pretend like these “business needs” don’t often boil down to a couple of tropes and that the story doesn’t change after a couple of post mortems are turned in for perfectly predictable failure modes.
teeray
Chatbots are great for this too. You get flippiness, but you also get auditing implicitly from the chat log.
jauntywundrkind
Two things,

1. The time to delivery is potentially much much shorter.

2. There's a built-in rules engine for targeting. You could integrate this in! But it feels nice having it separate.

danpalmer
It's good to have the current state of all the flags in source control, just like you do for things like infrastructure-as-code.

The distinction is that you have a different release process, or build a different artifact, from your main codebase. The codebase you are controlling with flags doesn't change when your flags do. This can be done with separate repos if you want one build per repo, but it doesn't have to be.

solatic
If you split feature flags into a different repository, then you're losing the benefit of having everything in a single repository, of having consistency and avoiding situations where your codebase refers to feature flags that don't exist, old feature flags that are no longer checked by the codebase, etc.

At this point, your production infrastructure is no longer solely one stateless server + one database, but two databases: your RDBMS and your GitOps repo tracking feature flags. Do you really get enough value from the second GitOps database compared to putting your feature flags in your main RDBMS?

danpalmer
With one repo you still have versions of this problem – have the flags been deployed as config to the flag system before the server build, has the server build removing usage rolled out fully by the time you remove the config. Using an RDBMS is an option, but makes scaling harder, you still need an audit trail, review processes, etc, so you eventually end up building a source control system on top of it (assuming you hit all the sharp corners and put time into solving them).

If you're feature flagging client code (i.e. somewhere you don't control rollouts, like mobile and web apps) that adds another layer of complexity.

While it's nice to have a simple system, having built one from scratch and used very mature feature flagging systems, my experience is that production systems hit almost all the edge cases quite quickly and flagging/experimentation systems are forced to evolve quite quickly to actually account for these issues.

Multi-repo or not isn't really an issue. My previous company had flag config in a separate repo, my current company has a monorepo, it doesn't really make a difference.

solatic
> Using an RDBMS is an option, but makes scaling harder

Modern Postgres scales vertically quite well on modern hardware.

> you still need an audit trail, review processes, etc,

But you need this anyway for the RDBMS in your architecture. You need an audit trail for when engineers need to get into the production database, and to show that their changes passed review, etc. My point is, if you anyway need to build this for your RDBMS, then you can build on top of that for your feature flag system if you throw that into your RDBMS as well.

> my experience is that production systems hit almost all the edge cases quite quickly and flagging/experimentation systems are forced to evolve quite quickly to actually account for these issues.

I think that's more an argument to use a commercial feature flag platform (like LaunchDarkly) instead of a FOSS option. A commerical platform is anyway what I would prefer to recommend! But, with the context of "choose a FOSS option", it seems to me like building on top of RDBMS, rather than GitOps, makes more sense.

danpalmer
Many services won't need an RDBMS, and for those that do there's a difference between the control plane of an application (development and releases), and the data plane (users using the service).

This is a complex and nuanced topic, but on my previous team of ~6 where we built a custom solution, we decided against using an RDBMS for multiple reasons, and on my current team where we use the same flagging system across 15 or so >1m requests per second services, there's no way it would work for us. If it works for your use case, that's great! But my advice for anyone else reading would be to put a lot of effort into considering the options as it's hard to change later and has significant impact on how the flagging system is used.

As for whether to use a commercial platform... my preference is probably to build my own with what I need in a system that I can modify as needed, or a commercial platform if there's one ready to go at a good price with the right feature set. I probably wouldn't use an existing open source option here unless I was forking it and treating it as my own from then on, as I find these things need flexibility and customisation. I've yet to see a great open source option.

thayne
It could probably be in a separate repository with access controls and/or deployment lifecycle.

In particular, a change to the yaml for feature flags could bypass most of your build and test pipeline, and changes could be deployed more quickly.

OTOH, you still need to figure out a deployment strategy for it.

mrweasel
I'd agree, it seems like the entirely wrong place to put a feature flag. Personally I'd go for either configuration file or database and then have a process for updating the feature flags.

The best implementation I've seen was in a Java project. Features where enable or disabled by either the properties file or the database. If a flag was set in the database, then that took precedence. New features would always be rolled out disabled in the properties file. Then in a controlled window the new features would be enabled for a few minutes and logs would be examined. If everything looked good the feature would then be enabled again. After a few days or weeks, the properties file would be updated to have the feature enabled by default and the flag in the database deleted in a later task.

miraantabrez
Agreed that you lose the benefit of "instant updates" if it's always controlled by a PR.

But Git-style version control with history, diffs, branches and pull requests are pretty useful for feature flags and other "app configuration".

Version history and diffs are great for knowing what flag logic changed when + debugging what broke prod.

Branches let you test and preview flag logic changes in your own isolated branch (which you can point the SDK at) — this is a cleaner approach to having a few separate "environments" like development, staging, production which can drift from each other.

Branches are also great for refactoring the schema / structure of all your flags, e.g. deleting a bunch of flags in one go.

Pull requests and approvals are great for when you're making changes to sensitive flags. E.g. you can lock down specific flags.

Pull requests are also great for onboarding nontechnical team members like PMs or sales reps so they can safely make flag changes themselves but require approval from an engineer (at least while they learn to use the system). Empowering nontechnical people is also why a UI is important.

Branching and pull requests are also a great way to prevent conflicts / overwriting other team members flag changes.

So Git-style features are pretty useful, but you also want the UI and you only want to enforce pull requests for specific flags or team members — this is what we built at Hypertune.

lnxg33k1
You can get the same comfort of fast updates and history with a feature flag system developed using event sourcing, without having the overhead of git
rekoros
If you’re looking for an open source feature flag option, also check out https://www.growthbook.io/
peab
I introduced feature flags at my last company, and ended up choosing growthbook. I love how compute experiment buckets with local hashing, so you don't have to make roundtrip server calls to get flag values.
ripperdoc
Not directly related to Dorkly, but we've implemented feature flags (with our own system) and found them not super-useful and was hoping for more - but we may be doing it wrong. I can certainly see feature flags working well for us when activating e.g. new mostly UI-related features, but when many services and APIs need to change in unison for new features it seems a lot harder to use feature flags in practice. Then it goes beyond just putting new feature code behind conditions, as you might need to load different dependencies, offer different version of server APIs, run on different database schemas, etc. But maybe we are missing something?
cqqxo4zV46cp
You aren’t. This doesn’t mean that feature flags are useless or not worth it, but there’s no silver bullet. It is exactly as hard an engineering problem as it sounds.
viraptor
You need to organise your features in a way that these are not a problem. Need different dependencies - load both. Need a different schema - write both versions, drop old later. Need new services/APIs - feature flip only the user-visible one.

The flags are really useful for things like enabling just a fraction of traffic, or ensuring you can switch a feature off much quicker than a full deploy would take.

hinkley
Everyone laughs when I say this but pull up a thesaurus. When you change the semantics of a thing, you have to change names to have old+new live at the same time. Don’t trust your mastery of English (especially if it’s your second language). There’s a synonym out there that describes the new behavior as well or even better.
waynepan
I'm also unrelated to Dorkly but I'm a big believer in feature flagging. It's key to moving fast as an engineering/product team.

be happy to walk you through how we use it. Shoot me an email if you want to chat. wayne at inaccord.com

hinkley
Last time I did this, we ended up routing our flags through a reloadable config system which used Consul for distribution.

We almost never shared flags across our fairly chunky servicees. We usually found a softer way to do it.

Even with Consul, you can still have enough skew that a few requests in the middle might see A and !A if your request rate is high enough. So it depends on your business model and architecture if that’s acceptable.

dagss
For systems with many services that need 99.9..% uptime, the ways to do ANY change is things like expand-contract.

In most such cases you have several instances of your backend running in parallel for scaling and redundancy and when making a release, instances of several versions run concurrently. So you don't have a "atomic upgrade" available

With multiple services coordinating upgrade is even harder.

Patterns like expand-contract helps you manage this..E.g. first add the new endpoint to server, then move clients over, then remove old endpoint.

So..feature flags is just a way of dragging this process over longer time period, and roll over % of traffic. Instead of coupling changes to service releases you roll over using config.

Used them a lot in backend to backend, backend to DB etc scenarios, has been hugely useful to us and would never work without it.

But, of course depend on context what you are doing.

[deleted]
(comment deleted)
theogravity
Neat! I'll look into integrating this with my feature flag abstraction library:

https://github.com/theogravity/feature-manager-wrapper

Edit: It looks like it's a backend replacement to LaunchDarkly, but you can still use the LD client from what I'm reading here, so there's nothing for me to integrate here.

icansearch
Similar to https://openfeature.dev/ ?
theogravity
Same space but I don't see any integrations listed at all on the site, just SDKs for someone to write their own integration.
esafak
I would expect an open source feature flag solution today to support https://openfeature.dev/
kageiit
Feature flags are great for safely releasing features fast.

As you add more of them though, they add tech debt and make the code harder to reason about. Developers are rarely motivated to clean them up after rollout.

_corym
My team leverages feature flagging heavily. Developers want to clean them up but product doesn't prioritize the tickets, and it creates more burden for QA as well having to do a full regression test.
goosejuice
Sounds like a great reason to include cleanup as part of the initial work.
willsmith72
The issue is not about feature flags then.

It's stakeholders not giving engineers necessary time for upkeep, probably caused by engineers lacking ownership, or failing to communicate

hinkley
It’s a system where engineers are signing off on work that isn’t actually complete. Which is a problem we had in Waterfall and mostly solved in Agile, except for feature toggles.
millerm
Then that is simply a bad lead on the team. "Do your job." (Not you)
triyambakam
In my experience engineers usually want to remove the outdated flags but product does not prioritize that work.
hinkley
I have found it very professionally unsatisfying to have a clear and documented reason I was unable to code up to my own standards. It doesn’t help of it’s somebody’s fault or everybody’s fault instead of just mine. It might actually be more upsetting, but I’d have to reflect more on that.

At the end of the day it’s still fuckery I was unable to avoid.

andruby
We had an experience where it was product that pushed for cleaning up the flags.

The best companies I've been at where those where product and engineering really operate as one team.

hinkley
At the risk of sounding like a broken record, I think this is a problem that is compatible with Kanban and incompatible with Scrum. The end of sprint encourages people to mark a task as done that still has several steps and about eight business days before it’s “Done”. But WIP limits in Kanban could be tweaked to account for this and I believe still provide the correct sort of pressure for you to finish what you started before starting something new.

In Limited Work In Progress (LWIP aka WIP-Limited) processes, if you are blocked on your story you help someone else get unblocked on theirs. And then escalate if there is no movement.

spike021
As someone whose current team process is Kanban and a previous team were also, tech debt like this still inevitably piles up. Sure maybe you can squeeze some time in but even getting a week to spend solely on cleanup is rare, if it comes up at all.
hinkley
The distinction I’m talking about is marking the feature milestone for deployment but not counting the feature as done until the toggle is dispensed with. Either removed, or declared a configuration we intend to keep indefinitely.

Admittedly, there’s still a gap there where a toggle may be needed until the end of an initiative (an Epic in Scrum terms) but that’s going to be at least an order of magnitude, and likely 2 orders, fewer toggles to deal with.

When the order of magnitude changes so does the solution.

keraf
At one of my first jobs, the codebase was massive and feature flags were heavily used. My first task given by my boss was to remove flags that have been enabled in all environments for a while. Great and engaging way to introduce someone to a new codebase, it's relatively easy and can be done with little assistance right of the bat, you also get to commit code in your first days and it reduces tech debt making everyone happy.

Been doing that with my new hires in other jobs.

miraantabrez
This is one of the reasons we built type-safe feature flags at Hypertune. When a developer deprecates a flag, they get type errors everywhere it's still used. This fails the build so they're forced to remove all the flag references, and they get a nice IDE / compiler driven workflow to do that. Otherwise it's easy to leave flag usages around.
alienchow
Nice, thanks for sharing. I've been looking for an alternative to CDPush (Google's internal version controlled config push).

Unaudited, untracked web UI based feature flagging has been my peeve with the other solutions. I'm surprised all the feature flagging solutions out there don't have this as a default. The next step would be to use an artifact versioning repository to decouple from S3.

wheresmycraisin
I have 25 years of coding experience, built two successful businesses, and I have no idea what this thing is supposed to do.
vyrotek
Think of feature flags as remote-toggleable IF statements.
TheCapeGreek
...which still begs the question - why does it need to be an entirely different service?

Probably answering my own question, but the main reason I can think of would be if your app doesn't have some kind of business admin panel capability.

I suppose my bias is also that in my sphere of web dev, we build these business panels pretty frequently, so feature flags and a UI for toggling them is something that makes sense to do within the app rather than add a third party service (and associated cost + potential latency) for it.

xmcqdpt2
You mean an admin panel on a server application, right? At my current job, I'm on a platform team that supports hundreds of applications. Each of them have their own admin panels.

However, there are settings that apply to all of them in our library code. For those, we use feature flags, and they are loaded from a network service, environment variables, code and config files. The overriding logic is complicated for legacy reasons and prone to bugs, so we are moving to a centralized feature flag system.

TheCapeGreek
Fair, that makes sense. Not a very common use case I think unless you're talking more about heavy use of microservices that need a feature flag across the entire suite.
jerrygoyal
i don't think most businesses need third party solution for feature flags. Just create an api and send config.
byyll
So feature flags without the UI?
drichelson
Well GitHub has a pretty simple UI for editing files + submitting pull requests, so let's say it's a different UI.
zellyn
Assuming their (also open source) Relay Proxy[1] can connect to this successfully, the disclaimers about lack of HA might not be too concerning.

We already run a giant pool of relays for our apps to connect to, both to save ingress/egress costs, and to provide a backup if LD blips out for a while.

[Edit: yep, it actually uses the Relay Proxy, in offline mode. See https://github.com/dorklyorg/dorkly/wiki/5.-Architecture]

[1] https://github.com/launchdarkly/ld-relay

zellyn
LD actually releases a crazy amount of stuff as open source. In addition to the Relay, the server SDKs are all open source too.

In fact, their entire API for flag manipulation and definition (essentially all the operations you can do with their UI, but programmatically) is open too: https://app.launchdarkly.com/api/v2/openapi.json

One could imagine running an OpenAPI → Go/Java/Rust/Python/etc. code generator on their OpenAPI spec, and then gradually implementing their entire API as open source.

Not sure how they would feel about that… although I suspect more usage of an LD-compatible API would only be good for them.

Of course, it makes sense for all those things to be open. The majority of smaller startups they're competing against will have settled on OpenFeature. DataDog too releases some equivalently surprising things as open source: eg. https://vector.dev/

drichelson
It's worth noting that there is no architectural limitation preventing an HA topology: multiple Dorkly servers can be deployed behind one or more load balancers for HA + lower latency where it matters (ie web and mobile apps).

If this is a limitation for anyone I'm happy to work with you on modifying the terraform module to support HA.

zellyn
Random question while you're paying attention here. In the example repo, I was unable to find a file that _didn't_ say "This file is managed by terraform. Do not edit manually."

Which yaml files is one supposed to edit to actually modify the flags?

drichelson
The example files are put in place by Terraform, so they will get overwritten next time `terraform apply` is run. You can of course change the example flag files as you're kicking the tires or testing out your app, but to really use the system you'll want to create new flags as documented here: https://github.com/dorklyorg/dorkly/wiki/3.-Common-Tasks#add... These new flags won't be managed by Terraform thus won't get reset to the default values.

This is good feedback. I'll clarify the comments

seattle_spring
Please choose a different name. It could be the most useful product in the world, but no way am I ever advocating something called "dorkly" at work.
zellyn
Ah, I've missed the CockroachDB name debate threads…
hbn
I was confused at first because I immediately recognized Dorkly as a YouTube channel that was very popular on YouTube around 2010, they produced comedy animations based on video games though I'm sure the humor isn't as funny as I thought it was when I was 13.

Checking now they have about 4 million subscribers. Probably not gonna be great for SEO longterm.

drichelson
Fair point. I’m open to suggestions for a new name :)
zellyn
I may or may not have grabbed the "LaunchOpenly" github organization name a while back. But I was intending to use it if anyone wanted to write an API-compatible Open Source clone of LD's APIs and flag-editing UI.