Launch HN: Airship (YC W18) – Controlled rollouts of new features
You may have seen that a million users are revolting over Snapchat's redesign, dropping their App Store rating below 2 stars. You might also remember when Digg launched a redesign in 2010 that caused a 30+% dip in usage in a matter of days. To avoid unpleasant surprises, companies like Facebook and Airbnb have developed their own sophisticated tools to carefully roll out their new features, test and measure their impact incrementally, and roll them back when signs of trouble appear. This way they can be informed about how users receive the features, and can fix mistakes without jeopardizing their entire user base. Then they can publicly release the new feature with reasonable confidence about its likely reception and effects.
Alvin and I discovered the importance of controlling feature launches and monitoring them carefully when we worked at Zenefits together. Many companies would like to be able to mitigate risk through controlled feature rollout and rigorous testing, but don't have the resources to do it. This approach is proven to work, but remains expensive, since only companies that can afford in-house tooling, and often a dedicated team to maintain it, can do it. It struck us that there was a need here for a product, so we started Airship to build it and make feature rollout accessible to everyone.
To use Airship, you install one of our SDKs. Then you can flag or gate code with if/else blocks calling a single function provided by our SDK. This automatically gets wired up to controls in the Airship dashboard which allows you to control targeting rules, rollout percentages, and whitelists/blacklists.
Key features of Airship: - Gradually roll out and A/B testing of entire features, not just copy and superficial changes - No performance hit (checking whether a customer has a feature takes < 0.1 ms) - 0% downtime (no dependency on Airship’s services, so effective downtime is 0%) - Complex experiments using attributes of Users, Groups (e.g. location, purchase amounts, device types, etc.) as well as other object types (Product Listings, Pages/Posts, etc.)
We'd love to hear your feedback and hear about your experiences, ideas, and needs in this area!
50 comments
[ 3.1 ms ] story [ 122 ms ] threadWhat you don't get out of those libraries alone is a couple things. One is a metrics pipeline if you want to actually A/B test and gather metrics on the success of different features. That's a hard problem because A/B metrics collection pipelines are something pretty much something everybody reinvents from scratch right now to fit their combination of language/logging tool/cloud host/data warehouse/BI tools/whatever.
The other big one is the managed configuration deployment - a UI that you can toggle features on and translate those to appropriately flipped toggles in all of your clients (mobile, web backend, web frontend, whatever). That's a pretty big task to do and requires a hosted service one way or another. Flipper (below) looks like it integrates into a rails service and so has a UI, which is kinda neat (first time I've noticed an OSS solution that does that part - looks like theres a number of options that do this now), but you'd still need to reinvent tooling for mobile apps and such with it.
[0]: https://github.com/jnunemaker/flipper [1]: https://github.com/disqus/gutter
https://github.com/checkr/flagr
https://github.com/Unleash/unleash
A few things:
1. More sophisticated targeting; i.e., different types of objects, combinations of attributes, and groups.
2. Airship's pricing model is aligned with customers' incentive to grow. Our architecture emphasizes local computation, so performance and cost scale very well (better than things like MAUs) - savings that we pass onto customers.
3. Airship will integrate with existing analytics tools (like Segment) so you have A/B testing capabilities without doing set up work. You can even set up an A/B test retroactively (after the feature has shipped), because Airship can annotate the events & metrics you already track.
4. Airship focuses on non-engineering teams as well (PM, Data, Growth), and making the dashboard intuitive for those use cases.
The pricing partially based on API calls feels somewhat vague though, it's tough to have any idea what an API call is in airship, especially given the abstraction level of the client where you explicitly remove the API from the sight of the developer. .1ms timings suggests it's not per flag, but then I see "Every time you check whether a User or an object should clear a particular gate". With that context 10,000 (or 75,000) seems very low - with 30 feature flags on a page you'd only be able to load 300 pages total across all users. It doesn't feel like I can judge how the API limits translate to real world usage, even though I do see the notes on the pricing page.
One thing I might recommend for the future as a killer feature is considering tooling around application performance between two cohorts. I've definitely seen cases where toggling a feature flag changes performance from 100ms to 800ms and nobody bats an eye because most small (and probably most large) organizations have crummy performance monitoring/tooling (PMs would probably love this too for projects with explicit performance goals).
What's the expected time in Airship for clients to pick up new configurations currently?
Couple questions
1. Is this another A/B testing framework? Or is it for feature enablement?
2. What level of granularity(user, cohort etc.) does the platform support?
2. Actually fairly granular, especially if you have properties or attributes you want to filter populations on.
This image serves absolutely no useful purpose (What is it?). It would've been much useful to see a screenshot of the product where you could set feature flags to at least give us an idea on what your product does.
We, at Repl.it, started using it for a very risky rewrite of a core part of our product -- the IDE (https://repl.it/site/blog/ide) -- and it helped us roll it out safely, turning it off when we needed to, when we get a user bug report Airship could quickly tell us if that user has the feature on, etc.
Since then we've been using it for virtually every feature release and experiment. We couldn't be happier with how smooth the service is:
- The service is reliable and fast.
- The UI is simple and instant.
- The SDK is simple and easy-to-use.
I know it's early days but maybe slightly simpler copy where you tell me what feature flagging actually is?
On the flip side, I learned something pretty cool today! :-)
One of the biggest draws for this sort of thing from the dev side is "how far am I going to have to bend over to use this".
Freely available documentation sounds like a no-brainer.
Here is the docs: https://airship.readme.io/v1.0/docs. We'll also link them in our home page in the next iteration.
You're probably already aware, but if not here's a heads-up that there's a company called Urban Airship that is in a related space (and based on what they do, your space could be in their sights). Just something to think about.
Good luck!
Congrats on the launch!
Regarding your pricing page: I feel the "10,000 API calls" and "75,000 API calls" limits on the Hobby and Growth plans would deserve some context. Is it 10k API calls per day or month? How fast do I exhaust it?
You comment mentioning that you emphasize local computation was helpful and the pricing page might benefit from including it.
For everyone else if you want to see their api's not very hard to take a look from their SDK.
https://github.com/airshiphq/airship-ruby/blob/master/lib/ai...
My initial questions are: - Will this work with frontend React code? - Do I need to create separate database hooks? - Can I create UI experiments without writing code? - Does it integrate with Stripe if we want to test conversions?