Launch HN: Haystack (YC W21) – Engineering analytics that don’t suck
I'm Julian, co-founder of Haystack (https://usehaystack.io). We’re building one-click dashboards and alerts using Github data.
While managing teams from startups to more established companies like Cloudflare, my cofounder Kan and I were constantly trying to improve our team and process. But it was pretty tough to tell if our efforts were paying off. Even tougher to tell where we could improve.
We tried messing around with JIRA which gave us story points and tickets completed but it didn’t help us dig deeper on where we could improve. We found a few tools that integrated with Github measuring # of commits, lines of code, and even comparing engineers using these metrics! - but we didn’t like that approach.
We wanted to know (1) how quickly we deliver as a team (2) what bottlenecks tend to get in the way and (3) as we make adjustments, are they helping us improve?
We scoured the internet looking for every piece of research on the topic we could find, talked to >500 engineering leaders working everywhere from startups to FAANG, and started to learn which metrics helped answer our questions and which ones just sucked. Once we had a clear picture of what that looked like, we built Haystack.
Haystack analyzes pull requests on the team level, giving you “northstar” metrics like cycle time, deployment frequency, change failure rate and 20+ more to help you improve delivery. Teams use Haystack to quickly find bottlenecks like code review, experiment with changes like smaller pull requests or automated tests, and see the result. Using this feedback loop, the top 70% of Haystack users have increased production deployments by 58% and achieved 30% faster cycle times on average.
We’re lucky enough to work with some awesome teams at Microsoft, Robinhood, and The Economist. As we continue to build out our product, we’d love to hear any of your experiences with engineering metrics, your thoughts about how to actually get them right, and of course your disaster stories :)
41 comments
[ 4.3 ms ] story [ 86.8 ms ] threadIn my line of work [1] I talk to a lot of developers about productivity, here are some questions about what you've built in the context of that experience:
1. I think (as you mentioned) "number of commits" or "number of pull requests" merged are proxies for productivity at best. Most teams that I've seen care more about "are the tasks for the epic being chipped down" - is that something your tool can help estimate as well? The progress bar for the epic? It'd be nice to get a table of the metrics and where they come from.
2. In the same vein, "scrum with epics" might not always be how teams decide on units of work, do you support other methodologies?
3. You mention deployment frequency a lot, but that seems tangent to developer productivity - Isn't getting things deployed quickly mostly for the benefit of the product folks? (getting feedback faster)
4. How does maintenance work fit into this? Does a developer who spends most of their time refactoring show up with "worse metrics" than someone pushing copy changes to the landing page?
[1] CEO @ https://layerci.com (YC S20)
1. While we do use pull requests as a unit of measure, we don't actually consider # of commits and # of pull requests as great proxies for productivity. We mainly track Cycle Time, Deployment Frequency, Change Failure Rate, and Throughput which help encapsulate speed vs. quality of an engineering teams. It's important for engineering metrics to correlate to engineering work. Specifically, that the engineering team has full control over those metrics. When looking from an Epic, Story, or Task perspective - this is often controlled by other parties, making it more difficult to know if engineering itself is improving or not. In addition to this, Epics often don't reflect actual engineering work in the same way that pull requests in Github do making them problematic to focus on.
2. Because we focus on engineering work itself from Github. Haystack can be used with any methodology so long as the team is using Git!
3. From an engineering perspective, the main goal is '# of successful iterations'. Product's main goal is that those successful iterations are pointed in the right direction. Looking at # of successful iterations (aka speed, deployment frequency, and quality) allow us to see how often we deploy value to customers and what is the quality of those deployments. Focusing on these metrics allow engineering teams to improve 'how they deliver'.
4. We actively don't look at individual engineers and we especially don't compare them using any metric since we've seen that lead to some bad takeaways like the one you mentioned. We help measure at the team-level to simply highlight improvements and bottlenecks rather than cross-compare any team or individual which is like comparing apples to oranges
https://www.usehaystack.io/#Pricing
The last company I worked at used the "Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations" book as their bible. It was a wonderful way to structure team rituals and review cycles.
But I suppose we will be building a Bitbucket integration shortly?
https://www.pluralsight.com/product/flow
I'm not sure if my public server can withstand it, but you can play with the public version at https://public-001.gitsense.com
If that doesn't survive, you can install GitSense (my product) with a single docker command, which you can find at https://gitsense.com
github integration sounds good for getting up to speed quickly and if I was still a CTO I'd be trying it out.
Mixed Content: The page at 'https://usehaystack.io/' was loaded over HTTPS, but requested an insecure favicon 'http://www.usehaystack.io/favicon.ico'. This request has been blocked; the content must be served over HTTPS.
Edit: It appears to be an issue on the naked domain only.
In the meantime, we've opened up our team Slack (link below) if you'd like more info on Haystack (or just want to jam about metrics with other eng leaders from companies like Microsoft and The Economist).
I believe this link should work: https://www.usehaystack.io/haystack-slack-community
And sorry for the technical difficulties. Should be resolved shortly.
I am trying to be more open minded about your work but I can't understand how developer productivity could be measured with any of these metrics. I'm not aware of any prior work or research that shows that either. Writing software is a complex task and quantising it in this way seems reductive and problematic.
I would welcome being wrong about this. What am I missing?
We've found that evaluating individuals, stack ranking teams, and measuring an individual's 'productivity' to be an ultimately futile effort.
It's more important to focus on team delivery as described in many research studies. My favorite being the book Accelerate by nicole forsgren. If you haven't, I do encourage you to check it out!
As the founder of Haystack I can be seen as biased so its probably more helpful to checkout their research across thousands of teams on what makes an 'elite engineering team' and how elite engineering teams ultimately help drive successful organizations across profitability, market share, and customer satisfaction :)
Edit: if you really got guts, let this post stand.
So you send alerts about burnout based on PR? We don’t need yet another alert. We need less alert. If this sounds crud, be it. If you have got guts, address these deficiencies head on.
I've banned this account and your other account in the thread. Breaking the site guidelines is not cool, and using multiple accounts to do it is especially not cool.
The webinar link is below - pretty cool to see how teams use metrics to really enhance the way they work. Even cooler to hear how happy the engineering team was after removing all those annoying friction points in the development process :)
https://www.youtube.com/watch?v=KrSc9EOrE_4