Rigorous? lol. I'm usually slightly critical of tesla.. but this article is terrible.
It's so long because the author needs to jump through too many hoops to end up where he wants to go. Audi A6 has 0 causalities? Cool. Tesla has 0 in one year? Extremely suspicious. Not enough Tesla accidents in the database? Include China. Should he include China for the other cars? Nah.
The data is in the article. They didn't like the conclusion, so then they included China, did a google search for tesla accidents, and made adjustments--only to tesla--until they got what they wanted. They aren't even comparing like with like... all of the conclusions are invalid.
I don't think this article was written in good faith.
They found clear Tesla fatalities that were missing from the published data, apparently a significant number of them due to miscategorization.
Now maybe all the brands have the same problem with missing data, but that would mean fatalities across the board are much higher than reported by the insurance agency funded research firm. That doesn’t seem as likely to me, but I do believe more research needs to be done.
I'm only seeing 1 fatality from China, does that move the needle substantially in this analysis?
> To have the largest sample size and best confidence intervals possible, we also included international accidents (three total, from Canada, Holland, and China). Given the higher incidence of fatal accidents per vehicle-year in China compared to the United States, and the lower incidence of fatal accidents in Holland and Canada, these three deaths should adjust to 3.63 deaths. However, for the sake of conservatism and to avoid a large debate about a rather small difference, we use 3.00 deaths in our analysis.
Yes, they are obviously biased and openly so, but they've raised what appear to be valid points that Tesla should answer. Elon made the claim publicly, and Tesla is no stranger to writing snatching rebuttals.
As disclosure, I'm long TSLA, both index and individual shares.
Yeah... even basic algebra requires that what you do to 1 side of the equation, you do for both and that definitely didn't happen here. There's no control at all in this analysis.
Is this really what passes for statistical analysis in the investment community? From a biomedical statistics background, this analysis so atrocious that I don't know where to start.
So please explain. It's quite a long story but with some good finds (data quality problems, obfuscated reporting, probability of much higher casualty rate). I wonder what your statistical rigour makes of this.
Well, including data from China for only Tesla is a joke. That would be like saying "we couldn't find any deaths from Ebola in the United States, so we took the ones in Africa and added them in. Wow, look at how many deaths per year from Ebola occurred in the US compared to deaths from the plague in the US!". It makes absolutely no sense.
Well it might be. They also include a casualty from NL at face value, while casualties are MUCH rarer here. I agree with the need for discounting of a Chinese casualty, but it's quite common practice to try to get more data points. That is good practice, not bad, and we only seem to disagree with the execution (is a China casualty 1 or .8?) In this case it wouldn't matter the base take-away much.
You have to at least account for differences in base rate of accidents.
The typical normalisation would take at least published accident total to synthesise data for the other cars.
(By using sales numbers of cars.)
What he did there is statistically unacceptable.
The medical equivalent would be to compare side effect profile of current best of line treatment done in general populace to results to your new treatment's phase 1 trial results which is done in healthy people only.
Then advertise that.
Regardless, can we have the raw data please? Are the rates even meaningful when the incident counts are low? Vehicle years are not the right unit as this counts cars that are not being driven. Actual range driven is the right unit.
I totally agree. But keep in mind the effect "doing the right thing" has on the effect to be estimated. I venture it will not change the gist of the story. Heck, I should contact the author and build a shiny app to demonstrate. Even discounting a Chinese death 50% would put Tesla in the ballpark double the risk of luxury cars. My thought would be that Tesla is more a supersport car (McLaren, etc) in risk (for some drivers, say, male, young or midlife, etc).
That one local death in NL, the guy did double the allowed speed on a stretch where I can hardly hit 20% above the allowed speed accounting for traffic lights and local situation. The one Tesla death with the young kid, the same. Don't put maximum traction hundreds of HP in male hands. We tend to use it (sometimes).
I suspect the correct use of Chinese data would make error bars overlap, exposing the BS.
Moreover, they should use the data where miles drive is available, say, from taxi drivers. Oops, that's 0 out of 0 total? :)
For such low incidences, even studentising will not produce right confidence intervals based on normal distribution. Poisson distribution cannot even be used, direct binomial instead, because it is likely that the error due to approximation will be important. (calculate from rule of rare events)
The biggest problem is that they go through all sorts of hoops to add unreported Tesla fatalities to the data, then compare those against unadjusted other fatalities and report that there are "more than Triple" apples (adjusted) vs orange (unadjusted).
The least they could do is run the same stats for both Tesla and another luxury brand (say, Audi), and then compare like to like.
maybe some censored data problems here? I'm interested in seeing a jupyter notebook analysis on driver fatality rates
Does anyone have experience doing survival analysis? I've read a lot of the documentation of CamDavidsonPilon's Lifetimes. maybe i'll give it a shot, seems like the type of thing people would already do to price car insurance
Tesla’s mortality rate (41 deaths per million vehicle years) is so much higher than the average luxury car (13 deaths per million vehicle years) that when comparing the two, the difference is hugely statistically significant. The difference is 28 additional deaths per million vehicle years, with a confidence interval of 11 to 63, and a p-value of 0.0001.
Binomial confidence intervals for individual cars except Tesla provided by IIHS
Binomial confidence intervals for Tesla and groups using Conflint.xls from John Pezzullo
so he is doing some kind of hypothesis testing, using the binomial distribution but I wish he would post the data and spreadsheets used to generate everything. probably some p-hacking going on?
I'm not really that statistically literate, just a computer engineering dropout if someone could explain that'd be cool
Written by a L/S HF analyst with a current position on TSLA: "Following this publication, we intend to continue transacting in the securities covered herein [...]"
The author of this article claims to have received an award for at least one published medical study, and I think someone should take a serious look at it. If he thinks that this methodology is okay in a study, I can't imagine what would happen if someone used his medical study for anything but laughs.
I submitted the OP, and am a bit surprised to see it flagged. The OP looks to me like a sincere attempt to analyze fatality data.
Yes, the authors are short Tesla, but they disclose it. Yes, the analysis is naive, but that doesn't necessarily mean its conclusions are off. Yes, the inclusion of data from China is... questionable, but the authors address it upfront: "To have the largest sample size and best confidence intervals possible, we also included international accidents (three total, from Canada, Holland, and China). Given the higher incidence of fatal accidents per vehicle-year in China compared to the United States, and the lower incidence of fatal accidents in Holland and Canada, these three deaths should adjust to 3.63 deaths. However, for the sake of conservatism and to avoid a large debate about a rather small difference, we use 3.00 deaths in our analysis."
Doesn't deserve the flag IMO, Tesla is an emotional topic for many people. While there certainly have been hit pieces and hack journalism throughout the companies existence, I think this one has enough meat to be worth discussing.
Like most investors, I'm long TSLA. Both index and individual shares.
I would have liked to see the HN community rationally debate what to me looks like an earnest but naive analysis of fatality rates -- based on its merits or lack thereof. Instead, at least some in the HN community have resorted to ad hominem attacks ("Tesla short!") and flagging.
For the record, I'm a huge fan of Musk and the company, but other than minimal long exposure through passive funds, I have no position in Tesla.
Means that they have short-selled Tesla stock. Simply put, they are betting on the Tesla share price to fall.
Edit: Together with the highly dubious "statistical" analysis they did, this COULD suggest that the article is intended to create doubts about Tesla and thus drive the stock price down. However, this is of course speculation.
32 comments
[ 2.9 ms ] story [ 70.0 ms ] threadIt's so long because the author needs to jump through too many hoops to end up where he wants to go. Audi A6 has 0 causalities? Cool. Tesla has 0 in one year? Extremely suspicious. Not enough Tesla accidents in the database? Include China. Should he include China for the other cars? Nah.
This article is a joke.
I don't think this article was written in good faith.
Now maybe all the brands have the same problem with missing data, but that would mean fatalities across the board are much higher than reported by the insurance agency funded research firm. That doesn’t seem as likely to me, but I do believe more research needs to be done.
The US had 12.9 fatalities per 100k cars. China? 104.5 -- 8 times higher.
Come on. This is a joke.
> To have the largest sample size and best confidence intervals possible, we also included international accidents (three total, from Canada, Holland, and China). Given the higher incidence of fatal accidents per vehicle-year in China compared to the United States, and the lower incidence of fatal accidents in Holland and Canada, these three deaths should adjust to 3.63 deaths. However, for the sake of conservatism and to avoid a large debate about a rather small difference, we use 3.00 deaths in our analysis.
They looked for Tesla fatalities because they needed more to make the argument they wanted. You understand the distinction, right?
As disclosure, I'm long TSLA, both index and individual shares.
This sounds like a good basis of a line-by-line refutation or counterargument on Medium. :)
What he did there is statistically unacceptable.
The medical equivalent would be to compare side effect profile of current best of line treatment done in general populace to results to your new treatment's phase 1 trial results which is done in healthy people only. Then advertise that.
Regardless, can we have the raw data please? Are the rates even meaningful when the incident counts are low? Vehicle years are not the right unit as this counts cars that are not being driven. Actual range driven is the right unit.
That one local death in NL, the guy did double the allowed speed on a stretch where I can hardly hit 20% above the allowed speed accounting for traffic lights and local situation. The one Tesla death with the young kid, the same. Don't put maximum traction hundreds of HP in male hands. We tend to use it (sometimes).
Moreover, they should use the data where miles drive is available, say, from taxi drivers. Oops, that's 0 out of 0 total? :)
For such low incidences, even studentising will not produce right confidence intervals based on normal distribution. Poisson distribution cannot even be used, direct binomial instead, because it is likely that the error due to approximation will be important. (calculate from rule of rare events)
The least they could do is run the same stats for both Tesla and another luxury brand (say, Audi), and then compare like to like.
Does anyone have experience doing survival analysis? I've read a lot of the documentation of CamDavidsonPilon's Lifetimes. maybe i'll give it a shot, seems like the type of thing people would already do to price car insurance
Tesla’s mortality rate (41 deaths per million vehicle years) is so much higher than the average luxury car (13 deaths per million vehicle years) that when comparing the two, the difference is hugely statistically significant. The difference is 28 additional deaths per million vehicle years, with a confidence interval of 11 to 63, and a p-value of 0.0001.
Binomial confidence intervals for individual cars except Tesla provided by IIHS
Binomial confidence intervals for Tesla and groups using Conflint.xls from John Pezzullo
so he is doing some kind of hypothesis testing, using the binomial distribution but I wish he would post the data and spreadsheets used to generate everything. probably some p-hacking going on?
I'm not really that statistically literate, just a computer engineering dropout if someone could explain that'd be cool
The author then went on google and tried to find more articles. Included whatever he could find only for Tesla.
If you're going to go through extra steps to pump Tesla's numbers, then of course they're going to be higher.
Duh.
Yes, the authors are short Tesla, but they disclose it. Yes, the analysis is naive, but that doesn't necessarily mean its conclusions are off. Yes, the inclusion of data from China is... questionable, but the authors address it upfront: "To have the largest sample size and best confidence intervals possible, we also included international accidents (three total, from Canada, Holland, and China). Given the higher incidence of fatal accidents per vehicle-year in China compared to the United States, and the lower incidence of fatal accidents in Holland and Canada, these three deaths should adjust to 3.63 deaths. However, for the sake of conservatism and to avoid a large debate about a rather small difference, we use 3.00 deaths in our analysis."
Like most investors, I'm long TSLA. Both index and individual shares.
I would have liked to see the HN community rationally debate what to me looks like an earnest but naive analysis of fatality rates -- based on its merits or lack thereof. Instead, at least some in the HN community have resorted to ad hominem attacks ("Tesla short!") and flagging.
For the record, I'm a huge fan of Musk and the company, but other than minimal long exposure through passive funds, I have no position in Tesla.
Edit: Together with the highly dubious "statistical" analysis they did, this COULD suggest that the article is intended to create doubts about Tesla and thus drive the stock price down. However, this is of course speculation.