Cause And Effect: A New Statistical Test That Can Tease Them Apart (medium.com)
"The key assumption is that the pattern of noise in the cause will be different to the pattern of noise in the effect. That’s because any noise in X can have an influence on Y but not vice versa."[...] "That’s a fascinating outcome. It means that statisticians have good reason to question the received wisdom that it is impossible to determine cause and effect from observational data alone." https://medium.com/the-physics-arxiv-blog/cause-and-effect-the-revolutionary-new-statistical-test-that-can-tease-them-apart-ed84a988e
57 comments
[ 2.5 ms ] story [ 154 ms ] threadLike many brilliant ideas, it seems so obvious in retrospect, another great "Why didn't I think of that?" moment.
See, for example:
Causation, Prediction, and Search by Spirtes, Glymour and Scheines (1993) https://www.cs.cmu.edu/afs/cs.cmu.edu/project/learn-43/lib/p...
Computation, Causation, & Discovery (1999), edited by C. Glymour and G. Cooper, MIT Press
That's a pretty big caveat.
Not saying the paper is not important, just that it's not (yet?) a full solution to the overall conundrum.
https://news.ycombinator.com/item?id=8777258
"that X and Y are dependent (i.e., PXY=PXPY), there is no confounding (common cause of X and Y), no selection bias (common effect of X and Y that is implicitly conditioned on), and no feedback between X and Y (a two-way causal relationship between X and Y)"
The fact that there is a common source of noise in both might make it more difficult to detect this asymmetry (because it weakens your characterization of A or B specific noise) but, on their assumption that noise is typically additive, wouldn't it still be theoretically possible to detect the asymmetry you're looking for? I.e., if A and B have common cause X, then noise from X will appear in A and B, but A noise will not appear in B, and B noise will not appear in A (assuming that A and B also have intrinsic noise sources in addition to the noise they inherit from X). Symmetry is maintained.
What am I not getting?
If you detect bleeding, then great, you have a result.
If you do not detect bleeding, then it could be that there is a confounding variable or it could just be that the noise doesn't bleed. Similarly if there is a confounding variable then you cannot guarantee that the noise will bleed to both sets of observations. You have know way of knowing. Make sense?
My understanding is that the more general case of causal discovery, where you have lots of variables, is already somewhat solved by Pearlian techniques (if you have A/B/C you can conditionalize on each and see what graph fits best - all independent, C confounding A & B, A -> B -> C, etc). But these techniques break down when it's just A and B: A->B and B->A look symmetrical - the problem is so simple that it's hard, because there's no C which can help break the symmetry and suggest something about the underlying causal graph.
OP is interesting because it points out that even without a C, A and B often aren't symmetrical in one particular way. So now pairs can be attacked as well as triplets and bigger; and maybe it points the way to new ways to tackle more complicated datasets.
"Statisticians have always thought it impossible to tell cause and effect apart using observational data. Not any more."
Maybe including "bivariate" in the paper's title could have helped.
This comes up often enough that it is a good case for linking related URLs together, which is something we intend to work on in the new year.
There is indeed an industry based on blowing scientific findings out of all proportion, and we do a lot to keep most of that at bay. But the solution isn't as simple as always linking to the papers, and there's no hope of keeping HN immune from this systemic dysfunction, only of correcting its worst excesses.
A high quality summary is important. For example, I would not grasp the significance (in the general case) of an HN post linking to an Arxiv paper titled "The entropy formula for the Ricci flow and its geometric applications". The summary will tell me that this is claimed to be (part of?) a proof of the Poincaré conjecture.
Also, causality in reality can be quite complicated if there are feedback loops: X-causes-Y-causes-X.
But, even if true, it isn't clear that this makes for a good test. For example, it's plausible that Y could have a damping effect and remove noise, which would reverse the results of the test.
"They say the additive noise model is up to 80 per cent accurate in correctly determining cause-and-effect." This has been exaggerated by Medium from "accuracies between 65% and 80%" in the original article.
But a coin-flip model should be 50% accurate. 65% accuracy is unconvincing. The journal article's conclusion admits that their results are not statistically significant in any sense. As such, the results do not even meet the weakest possible scientific standard. They couldn't reproduce earlier published results in this field (typical of publication bias).
Their final paragraph concludes that there is surely a method of doing this, but they just haven't found that method here.
In my opinion, the results do not support that conclusion.
I don't know how hard it would be to collect more examples: on the one hand, any randomized experiment should be a source of such examples, but on the other hand, you might want datasets large enough to definitively establish a causal effect before you go using it as a ground-truth/benchmark in statistical studies like this (and larger datasets tend to be harder to get the individual datapoints for).
It would still be a good test of bivariate causation even if it doesn't handle the case of feedback well.
> The journal article's conclusion admits that their results are not statistically significant in any sense.
...and in the same paragraph, they point out that a standard Bonferroni correction would be inappropriate and very conservative, because many of the methods are very similar & the results correlated, and that almost all the variant methodologies (except 1 or 2) do >50%. (If you really want to be a smart-ass about this point, please calculate the odds of almost all the variants performing >50%.)
By my reading, they don't apply Bonferroni, nor any other correction for multiple-testing. The results are already not significant before the correction, so there is no point in making a correction step. Read this part again: they are saying their results are insignificant and would be further away from signficance if they had bothered to apply any correction.
Criticising Bonferroni as being too conservative for this purpose might be fashionable, but it's irrelevant and highly misleading here (neither is it evidenced). They haven't applied Bonferroni and they haven't even identified an appropriate replacement - this is an irrelevant comment which seeks to distract.
Give me a break. Let me go back and quote from the paper:
"Whether the performance of a method is significantly better than random guessing is not so clear-cut. For example, the p-value for ANM-pHSIC under the null hypothesis of random guessing is 0.068 on the CEP benchmark, only slightly larger than the popular α = 5% threshold. However, because we tested many different methods, one should correct for multiple testing. Using a conservative Bonferroni correction, the results on a single data set would be far from being significant. In other words, from this study we cannot conclude that the ANM-pHSIC method significantly outperforms random guessing on the CauseEffectPairs benchmark. However, the Bonferroni correction would be overly conservative for our purposes, as many of the methods that we compare are small variations of each other, and their results are clearly dependent. In addition, good performance across data sets increases the significance of the results. Although a proper quantitative evaluation of the significance of our results is a nontrivial exercise, we believe that when combining the results on the CEP benchmark with those on the simulated data, the hypothesis that the good performance of the methods ANM-pHSIC, ANM-HSIC, ANM-PSD and ANM-MML is only due to chance is implausible."
A p-value of 0.06 is not horrible, and they do not say it is the smallest p-value obtained. And since they test multiple methods, in different conditions, Bonferonni would divide 0.05 by combinatorially many, which is absurd because clearly these are not independent p-values (which is the assumption behind Bonferroni) and it is not irrelevant at all (and you saying it is suggests you don't understand how Bonferroni works).
In fact, you're not even right about "The results are already not significant before the correction" - ANM-pHSIC was almost statistically-significant at the normal threshold, and it didn't perform the best on the full unmodified dataset, that was ANM-HSIC (see figure 11, pg28, left table; or pg27, figure 9).
A more reasonable interpretation would be, rather than shoehorning in a Bonferroni where it has no business, to ask whether there's an overall difference of the various algorithms from baseline chance of 50-50, with something like an ANOVA.
How would that work? Well, we can estimate the fraction right by eye, use the known 88 pairs, and estimate from there. (There's probably a table of exact results somewhere but I didn't see it.) What is the p-value of all the algorithms, lumped together? Does it suggest outperformance? Why yes it does:
R> cep <- read.csv(stdin(),header=TRUE) "Type","Fraction" pHSIC,0.62 HSIC,0.65 ent1sp,0.45 entsp1,0.62 entKDP,0.70 endPSD,0.67 end3nn,0.20 entEdE,0.55 FN,0.49 Gauss,0.51 MML,0.57 entME1,0.60 entME2,0.55
R> cep$Correct <- round(cep$Fraction * 88) R> cep$N <- 88
R> binom.test(sum(cep$Correct), sum(cep$N), p=0.5)
Exact binomial test
data: sum(cep$Correct) and sum(cep$N) number of successes = 633, number of trials = 1144, p-value = 0.0003429 alternative hypothesis: true probability of success is not equal to 0.5 95 percent confidence interval: 0.523970021 0.582398490 sample estimates: probability of success 0.553321678
So even including all the crappy algorithms, they still outperform. An ANOVA would show the same thing. As the authors say: "the hypothesis that the good performance of the methods ANM-pHSIC, ANM-HSIC, ANM-PSD and ANM-MML is only due to chance is implausible."
http://google-opensource.blogspot.com/2014/09/causalimpact-n...
And their related research into using Bayesian structural time-series models to infer cause and effect.
http://research.google.com/pubs/pub41854.html
Can someone explain how this is 'obvious'?
How can this be a claimed scientific way to tell cause and effect and then drop a sentence like that in the middle of the explanation?
Even if you accept that it's true that temperature determines snowfall, it seems there is likely some feedback loop in there. The fallen snow doesn't just disappear, wouldn't it affect later measured temperatures? Remove a bunch of (cold) snow from an area and the average temperature of the area should increase faster than if you had left the snow, no?
However, in many real situations on earth, the temperatures often fluctuate right around the freezing point of the water in the space.
Unless your weather station is totally covered by snow, no.
And the effect of snow in the total thermal capacity of a city is small. Just compare its mass to the mass of "all things" (buildings, etc)
I'm interested in how things diverge at all scales.
Imagine two identical snowing days, except on one of the days you remove the 6 feet of snow on the mountain. There wouldn't be any difference in the amount of snow that falls? Or the local and global temperatures?
I think the mechanisms which set rental prices in a market, and the availability of properties of different sizes to students at different prices, are probably pretty complex functions of supply and demand factors that, themselves, are dependent on behaviors in other markets, like the mortgage market, the higher education loan market, the job market, and so on. You don't even need to talk to an economist; a realtor will tell you the most important factor driving housing costs: location, location and location. That's why two apartments of the same size in New York and Nebraska will have different rental rates.
Is price set by the supplier or the buyer? I think an economist would tell you 'both'.
The main problem is that it requires making assumptions that are very hard or impossible to test. Nonetheless it's an interesting idea, but I doubt this method can replace randomized trials or instrumental variables except in a tiny fraction is cases
> Concluding, our results provide evidence that distinguishing cause from effect is indeed possible by exploiting certain statistical patterns in the observational data. However, the performance of current state-of-the-art bivariate causal methods still has to be improved further in order to enable practical applications.
(I'm joking of course, but has anyone ever actually rigorously analyzed quantum random data?)
Looks pretty, shall we say, wacky, but they are supposedly finding correlations in quantum RNGs.
This isn't obvious to me at all.
It's true that rainfall causes trees (and that drought can kill them). But it's less obviously true that trees (in massive numbers) can affect the regional climate enough to cause rain. They do this by pumping water out of the ground and increasing humidity, by changing wind patterns, etc.
When trees cause rain, it's a lesser effect than when rain causes trees, but it's still there.
So when someone tells me that it's obvious that hundreds of thousands of tons of frozen, powdered water laying on the ground doesn't cause the temperature, I have to wonder if they're a retard.
What people really want to know is whether statistics alone can be used to exclude hidden shared causes from an uncontrolled data set. Even the article itself uses such an example: the impact of hormone replacement on heart disease.
This test does not further that goal. I remain convinced that it is impossible. In fact to my understanding, that is the origin of the scientific method: rather than accepting conclusions from the first data set, science constructs hypotheses and tests to exclude hidden causes.