The original analysis was done with the standard method and found a very strong signal.
A new analysis done with a non-standard method found a significantly weaker signal.
The argument for the non-standard method is that while the standard method substantially reduces noise, it also can be heavily affected by instrumental errors. They looked at more of the ALMA data and found other high signal to noise peaks which aren't associated with any chemicals and thus are probably artifacts from the detector. A different curve fit that gets rid of these erroneous peaks also shrinks the phosphine peak below standard statistical significance.
The problem though is that they only looked for these alternative peaks in the datasource for one detector, but the phosphine peak was found on multiple detectors. Overfitting may make a random peak stand out more than it should, but that peak has to be there to begin with. Either both detectors have some unknown quirk that causes the same blip at the phosphine line, or that's a real signal.
The argument that the signal is less than what is typically considered significant under the alternative analysis technique is irrelevant as the typical case doesn't use the alternative technique. If you changed all other measurements to reduce signal to noise ratio, the required signal to noise ratio for all detections would also decrease.
It is also worth noting that this new paper, which uses a non-standard method, has not yet been peer reviewed.
Thanks! This brings a lot of context to the table.
I have very little sense of how much the two detectors should add to the credibility of the finding. It seems like it would depend on the causes of instrumental bias and how much they vary among detectors.
However, I can definitely say that modeling background with a 12th order polynomial fit is not going to be robust. Any competent statistician would have said so if consulted. That it is the standard method among astronomers is just depressing, not at all confidence-boosting. And I don't think that just reducing the threshold when a more robust method is used will yield equivalent results.
Hopefully this prompts a thorough review and overhaul of astronomy's parochial and outdated approach to statistical data analysis.
2 comments
[ 4.6 ms ] story [ 18.2 ms ] threadA new analysis done with a non-standard method found a significantly weaker signal.
The argument for the non-standard method is that while the standard method substantially reduces noise, it also can be heavily affected by instrumental errors. They looked at more of the ALMA data and found other high signal to noise peaks which aren't associated with any chemicals and thus are probably artifacts from the detector. A different curve fit that gets rid of these erroneous peaks also shrinks the phosphine peak below standard statistical significance.
The problem though is that they only looked for these alternative peaks in the datasource for one detector, but the phosphine peak was found on multiple detectors. Overfitting may make a random peak stand out more than it should, but that peak has to be there to begin with. Either both detectors have some unknown quirk that causes the same blip at the phosphine line, or that's a real signal.
The argument that the signal is less than what is typically considered significant under the alternative analysis technique is irrelevant as the typical case doesn't use the alternative technique. If you changed all other measurements to reduce signal to noise ratio, the required signal to noise ratio for all detections would also decrease.
It is also worth noting that this new paper, which uses a non-standard method, has not yet been peer reviewed.
I have very little sense of how much the two detectors should add to the credibility of the finding. It seems like it would depend on the causes of instrumental bias and how much they vary among detectors.
However, I can definitely say that modeling background with a 12th order polynomial fit is not going to be robust. Any competent statistician would have said so if consulted. That it is the standard method among astronomers is just depressing, not at all confidence-boosting. And I don't think that just reducing the threshold when a more robust method is used will yield equivalent results.
Hopefully this prompts a thorough review and overhaul of astronomy's parochial and outdated approach to statistical data analysis.