This is just one of the reasons why I found FB infrastructure to be great. I used to work at Amazon before moving to FB and the difference in internal tool quality is night and day.
FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest hardware request ever.
So it should be no wonder that even tests are measured. Flaky tests get disabled. That's nice and keeps trust high in the test framework. The downside is that you are likely to forget to fix the disabled test, so now some functionality that was important enough to test in the first place becomes untested and leaves room for others to introduce regressions. This has happened in my team, but the pace at FB is so quick that there is no way to tie up all these loose ends without giving up your personal time. Or maybe it requires someone more diligent than me.
Either way, I just wanted to share that I personally found the idea of measuring test runs and acting on insights from thise measurements really powerful. It's a theme that can be found at all levels of FB infrastructure.
> FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest hardware request ever.
This is a natural consequence and trade-off of having a top-down directive of which tools/languages/frameworks to use. That just doesn't happen at Amazon.
Not sure what you mean. There is no hard and fast top down enforced rules. Aside from natural Hack bias at FB, you can use any language you want. People develop tools in Rust, D, C++, bash, etc.
As far as libraries go, there is a mandate to use in-house patched versions of various sdks and libraries, for security/customization reasons. Otherwise, go nuts.
People write a whole lot of in-house software at FB, some of which is later open sourced.
> That just doesn't happen at Amazon
Oh please. We had a mandate to use Apollo.
PHP is banned at Amazon (whatever your view of PHP is, that is a top down directive).
The only freedom we actually had at Amazon was to use whatever open source software we could without contributing back to it (I'm sure there are examples where we did, ut not at the scale of FB).
And lastly, Amazon was all about telling me what to do. One day I got a phone call and I got moved from front-end app development to backend c++. Nobody asked me if I wanted to do it or if I would be any good at it.
FB spends months in bootcamp with teams trying to convince you to join them.
> FB approaches internal code development with the same data driven rigor as their business decisions.
How does this even matter if the end result is buggy for years and there’s nobody to alert? It seems like Facebook has many versions of code that get deployed in some places and not others, never attempting to reach an eventual convergence even for the same feature.
I have a few anecdotes where, as a Facebook group administrator of some large (tens of thousands of members) and small groups (few thousand members), there were so many bugs in moderating the group with an added mega bug where the “Report issue to Facebook” (or something to that effect) would also throw an error. Those bugs lasted years (going into the current year) before I gave up and quit Facebook.
Based on my experience over several years, I doubt there’s much rigor on many things. It may just be one of those marketing messages that have a little bit of truth and a lot of subterfuge.
> tie up all these loose ends without giving up your personal time.
so this just means they've crammed more into your backlog than you can do. I would not give up any personal, unpaid time to do it, unless there's some promotion that you need which this is a demonstration/investment for.
Seeing as I made the backlog, I have nobody but myself to blame :)
I know what you mean though. Our deadlines are self-imposed, but the planning process is there to make sure that the effort remains high.
I believe the managers are responsible for squeezing out as much juice as they can in a polite way (as opposed to other places where they were pretty direct about it).
I suppose the combination of opportunity + compensation justifies the stress level and excitement at this point in my life.
I've worked at other places where people primarily coasted with low pay but received good retirement benefits. It seemed to work for them.
I kinda miss working on teams where disabling the tests was plan B instead of plan G. The disabled tests should hopefully reduce code coverage. Also someone should be graphing out the volume and flux in disabled tests from week to week, to tap the brakes if it looks like we're rushing forward blindly instead of doing our due diligence.
And will very quickly humble even the most diligent of testers :) For example, you're testing that this value is in a range - but are you testing both endpoints and whether they're inclusive or exclusive? In faster languages I enjoy quickcheck style testing which also reveals these things pretty quickly.
Do not confuse end-to-end functional tests that this article is describing for smaller “unit” tests that you typically might express in quick check/mutation testing (at least as I understand the space).
The final test is always human review. The point of testing is to simplify the analysis of correctness for humans.
Also, ideally your code and test are somewhat independently written, whether it is whitebox or blackbox. If that is the case,you greatly reduce the chance that both the code and the test are wrong in such a way that the test still passes.
This is why writing unit tests for stuff like getters and setters is really dumb. It doesn't really simplify checking whether the code is correct unless you use something like meanbean and thus provides some degree of separation between the test and the implementation.
I've found the number one cause of flakiness in tests is from misuse of nondeterministic or highly stateful functions. This becomes especially apparent when you recognize a lot of nondeterminism comes from misusing datetime libraries such as momentJS, or from Math.random . Even integration tests against certain "eventually consistent" databases tends to result in flakey tests. Entropy and Time libraries are intentionally nondeterministic, so the use of these should be recognized when writing tests, and designing functional "cores" that can do predictable operations when given testable inputs.
Yes, definitely that. I've also seen entire piles of tests be flaky because:
- They relied on something or other over the network, or
- They used Selenium to try to see whether the Dom was updated correctly, and either Selenium itself or the way the code interacted with it made _all_ the tests flaky to some degree.
The general trend I've seen is that the more "e2e" a test is, the flakier it is.
e2e is the number one productivity killer at my org, by a long shot. If there is such a thing as a non-flaky e2e test, I have yet to see it. That, or the test does nothing.
Selenium seems fine for the most part. It's a solid tool. Where it falls apart is when developers do not account for all the various ways the browser session will go wrong. There are simply too many variables at play. A/B tests, cookies, popups, network conditions, machine speed (and current load). Writing e2e is like being blindfolded and tying one hand behind your back while your coworkers take turns spinning you in your chair.
It constantly amazes me that companies think they can put more tasks on a developer's plate with zero impact to productivity. They really do believe that all testing is free and will somehow pay for itself. And yet a single QA human going through a simple testing plan will catch more actual bugs than hundreds of e2e tests that cost a literal fortune to maintain.
IME the source of flakiness in tests is always something that you can deal with given enough time and about 15-20% of the time it is a bug in the code itself, sometimes quite a dangerous one.
End to end tests definitely end up being flaky, especially in large systems. One level of testing would be unit tests, but e2e tests have their own place, where they do end to end sanity checks. In my experience at Rippling, we have managed to identify a lot of such flakiness by pure first principles reasoning of the behavior, and in most cases, it turned out to be a subtle bug in the code. As the org grows larger, there should be a team that just attacks flaky tests, either from a fix point of view by reviewing tests, or from tools point of view, where finding the gap becomes easier for the product teams!
But sometimes the bug is in the code not the test, and you wouldn't have known about the bug if you didn't write the flaky test! A flaky test which fails once in every N test suite runs is better than no test at all.
We actually have different classes of tests to allow for some more flaky tests. You definitely don't want to run those flaky tests after every build, but you should be able to eventually get a run where those tests pass before handing it off to customers.
They say that contempt is the beginning of the end of the Rule of Law, which is why you should be careful not to pass frivolous laws.
Tests I've found are much the same way. You don't write one flaky test and stop. If you write one and everybody is okay with it, you and your coworkers write more, and more, until there are 50, 100. Once the suite flakes out on an interval, nobody takes a failed test seriously, and then broken code doesn't get checked for hours. One broken test? I bet it's the usual. I'll just rerun it a couple of times.
The thread you're pulling on here starts to unravel the whole Continuous Integration sweater.
The worst thing to deal with in regard to determining correctness of code is global, shared, mutable state. Timestamps fit that bill (even though it's the Universe changing the state).
For testing purposes I often find myself making functions that take the date as an argument. If the language supports default values, I'll set it as a default value. If it doesn't, I'll make a convenience method or a null check to set it to 'now' if none is provided.
It turns out though that a lot of code we write to run within ±30 seconds of 'now' ends up over time having to run (or re-run) on old or future dates. So with the exception of logging and events, having that as an argument turns out to be useful or at least neutral.
For logging and events I'd probably use a mock timing library anyway.
This is a great example of using statistical inference to solve a problem at scale. However I had one question: Since this using Bayesian inference, how was the prior over the parameters p_f and p_b determined? I could not tell from reading the article.
> While engineers tend to trust passing test results, they often retry failing tests a number of times on the same version of code and consider failures followed by passing results as flaky. We do not have a good theoretical explanation for why this behavior prevails.
which seems to me like standard confirmation bias.
When writing a test or updating code, you clearly assume you got it right. So a passing test “confirms” that.
Personally, I find myself quite susceptible to this bias when updating code (i.e., when I haven’t broken any tests, I assume I updated my code correctly) but I’m much more cautious when writing new tests. I usually prefer to write a failing test, and get it to pass, to mitigate some of that bias (but fully admit that once it turns green, I’m overly confident at the “clearly correct” nature of the output).
How is measuring or estimating flakiness different than measuring or estimating variance? To me, something that is not flaky never errors for no reason. Something that is flaky errors infrequently and not obviously why. Something that is broken errors a lot. So, to me, flakiness is stochastic variation in a recurring process /with/ some ambiguity about the sources of variation. Flakiness make sense to me as a term for describing variation in relation to cause. That's different than just variance - or maybe flakiness is just variance? I am wondering what the FB framework looks like from a statistical process control POV.
35 comments
[ 2.7 ms ] story [ 103 ms ] threadFB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest hardware request ever.
So it should be no wonder that even tests are measured. Flaky tests get disabled. That's nice and keeps trust high in the test framework. The downside is that you are likely to forget to fix the disabled test, so now some functionality that was important enough to test in the first place becomes untested and leaves room for others to introduce regressions. This has happened in my team, but the pace at FB is so quick that there is no way to tie up all these loose ends without giving up your personal time. Or maybe it requires someone more diligent than me.
Either way, I just wanted to share that I personally found the idea of measuring test runs and acting on insights from thise measurements really powerful. It's a theme that can be found at all levels of FB infrastructure.
This is a natural consequence and trade-off of having a top-down directive of which tools/languages/frameworks to use. That just doesn't happen at Amazon.
As far as libraries go, there is a mandate to use in-house patched versions of various sdks and libraries, for security/customization reasons. Otherwise, go nuts.
People write a whole lot of in-house software at FB, some of which is later open sourced.
> That just doesn't happen at Amazon
Oh please. We had a mandate to use Apollo.
PHP is banned at Amazon (whatever your view of PHP is, that is a top down directive).
The only freedom we actually had at Amazon was to use whatever open source software we could without contributing back to it (I'm sure there are examples where we did, ut not at the scale of FB).
And lastly, Amazon was all about telling me what to do. One day I got a phone call and I got moved from front-end app development to backend c++. Nobody asked me if I wanted to do it or if I would be any good at it.
FB spends months in bootcamp with teams trying to convince you to join them.
That's not a natural consequence of the structure.
If the top-down directives are issued by competent and financially accountable leadership, you get a nice efficiency pipeline.
How does this even matter if the end result is buggy for years and there’s nobody to alert? It seems like Facebook has many versions of code that get deployed in some places and not others, never attempting to reach an eventual convergence even for the same feature.
I have a few anecdotes where, as a Facebook group administrator of some large (tens of thousands of members) and small groups (few thousand members), there were so many bugs in moderating the group with an added mega bug where the “Report issue to Facebook” (or something to that effect) would also throw an error. Those bugs lasted years (going into the current year) before I gave up and quit Facebook.
Based on my experience over several years, I doubt there’s much rigor on many things. It may just be one of those marketing messages that have a little bit of truth and a lot of subterfuge.
Whether an internal team uses those tools effectively depends entirely on them and their expertise.
so this just means they've crammed more into your backlog than you can do. I would not give up any personal, unpaid time to do it, unless there's some promotion that you need which this is a demonstration/investment for.
I know what you mean though. Our deadlines are self-imposed, but the planning process is there to make sure that the effort remains high.
I believe the managers are responsible for squeezing out as much juice as they can in a polite way (as opposed to other places where they were pretty direct about it).
I suppose the combination of opportunity + compensation justifies the stress level and excitement at this point in my life.
I've worked at other places where people primarily coasted with low pay but received good retirement benefits. It seemed to work for them.
Also, ideally your code and test are somewhat independently written, whether it is whitebox or blackbox. If that is the case,you greatly reduce the chance that both the code and the test are wrong in such a way that the test still passes.
This is why writing unit tests for stuff like getters and setters is really dumb. It doesn't really simplify checking whether the code is correct unless you use something like meanbean and thus provides some degree of separation between the test and the implementation.
Ain't no one got time for flakey tests.
- They relied on something or other over the network, or - They used Selenium to try to see whether the Dom was updated correctly, and either Selenium itself or the way the code interacted with it made _all_ the tests flaky to some degree.
The general trend I've seen is that the more "e2e" a test is, the flakier it is.
Selenium seems fine for the most part. It's a solid tool. Where it falls apart is when developers do not account for all the various ways the browser session will go wrong. There are simply too many variables at play. A/B tests, cookies, popups, network conditions, machine speed (and current load). Writing e2e is like being blindfolded and tying one hand behind your back while your coworkers take turns spinning you in your chair.
It constantly amazes me that companies think they can put more tasks on a developer's plate with zero impact to productivity. They really do believe that all testing is free and will somehow pay for itself. And yet a single QA human going through a simple testing plan will catch more actual bugs than hundreds of e2e tests that cost a literal fortune to maintain.
Tests I've found are much the same way. You don't write one flaky test and stop. If you write one and everybody is okay with it, you and your coworkers write more, and more, until there are 50, 100. Once the suite flakes out on an interval, nobody takes a failed test seriously, and then broken code doesn't get checked for hours. One broken test? I bet it's the usual. I'll just rerun it a couple of times.
The thread you're pulling on here starts to unravel the whole Continuous Integration sweater.
For testing purposes I often find myself making functions that take the date as an argument. If the language supports default values, I'll set it as a default value. If it doesn't, I'll make a convenience method or a null check to set it to 'now' if none is provided.
It turns out though that a lot of code we write to run within ±30 seconds of 'now' ends up over time having to run (or re-run) on old or future dates. So with the exception of logging and events, having that as an argument turns out to be useful or at least neutral.
For logging and events I'd probably use a mock timing library anyway.
> While engineers tend to trust passing test results, they often retry failing tests a number of times on the same version of code and consider failures followed by passing results as flaky. We do not have a good theoretical explanation for why this behavior prevails.
which seems to me like standard confirmation bias.
When writing a test or updating code, you clearly assume you got it right. So a passing test “confirms” that.
Personally, I find myself quite susceptible to this bias when updating code (i.e., when I haven’t broken any tests, I assume I updated my code correctly) but I’m much more cautious when writing new tests. I usually prefer to write a failing test, and get it to pass, to mitigate some of that bias (but fully admit that once it turns green, I’m overly confident at the “clearly correct” nature of the output).
Yet again, a problem is not solved but paved over by a new level of indirection, and the house of cards grows taller.
They can't help it, each facebook engineering blog post has to start with that scale porn.