So basically they took a simpler version of software used in hundreds of medical imaging labs around the world and put it in an app?
I've worked in labs that do this exact same thing. It's cool that they're putting this in a consumer-facing format but the technology (re: accuracy in ML models) still isn't there yet, and there are a lot more performant and sophisticated models than the relatively simple network architectures the authors describe. At risk of being overly critical of the author, to me this just seems like a first, rather naive implementation of ML to be shown as a proof of being able to do ML related things, with the actual performance or usefulness of the model being secondary to that.
In short, putting this into practice at a consumer level: good. Actual modelling: maybe not so good.
> So basically they took a simpler version of software used in hundreds of medical imaging labs around the world and put it in an app?
> I've worked in labs that do this exact same thing. It's cool that they're putting this in a consumer-facing format but the technology (re: accuracy in ML models) still isn't there yet
Don't these observations explain each other? The whole point of have a consumer product is to reduce office visits and/or increase the monitoring frequency. So reduced accuracy is acceptable since any positive results would be followed up by the standard, more sophisticated tests.
They aren't trying to mimic the state of the art in a phone.
Medical diagnostics have to be held to a high standard. False negatives (the app says you're fine when you have cancer) and false positives (the app says you have cancer when you're fine) both have significant consequences.
False negative (24%) means you are not getting the treatment you need as soon as you could. Imagine people delaying a real mammogram for a few years because they have an app that says "all good!".
False positive (35%) means unneeded tests and doctors visits. This is less concerning as a bad outcome, but it does mean more stress , more expense and longer wait times to schedule a doctor visit.
This is an interesting exercise in ML, but there is little chance this app would get approval as a diagnostic tool.
I'm not making any claims about the absolute accuracy of the test or whether it's useful as is. I'm responding to the criticism that (1) the test is less accurate than a lab test and (2) who cares about putting it in a smart phone.
Also, FTA:
> The code will be the deliverable to iSono Health as a baseline model for further algorithm development.
Yeah, considering one presumes this is a screening test, the sensitivity is a catastrophe: a negative result means you have no idea if you have cancer or not. You spent money that could have been better spent on just about anything else (composting the dollars, for instance).
Well, like I said there is some research suggesting classsification on mammo + US can perform better than mammo alone. Hardly an unusual result in ML work.
The data should have been split into train, validation, and test sets before augmentation was applied. As it is now, there are likely examples in the validation and test sets that are highly similar to examples in the training set. The goal of validation should be to measure performance on previously unseen patients, not previously unseen scans of patients you've already trained on.
Because the data augmentation is performed prior to the random train/val/test split, nearly identical instances may be found in each set. In other words, a 1 degree rotation of training image X may be found in the validation or test set which would artificially inflate performance.
Getting image recognition to work to "replace" radiologists is both a game-changer and a HUGE money-saver and thus a GIGANTIC opportunity. Radiology is the most common specialty and one of the highest-paid in just the US -- 40,000+ radiologists earning $350k/year (so maybe $15 bn in salaries). Radiologists and thus some of the smartest & most capable people in the country and they spend the whole day just looking at images...computers should be doing this!! Not only that, computers will soon better...for many skin tumors and others, even the leading human experts differ in their opinion, so humans today out-perform computers but I don't foresee that being the case for long.
A game-changer in cancer treatment would be "early detection". Once it gets to be inexpensive to analyze images, then we can have vastly earlier cancer detection...you could work into a scanning machine in a local pharmacy, get scanned & get the analysis right there...you could go in for monthly scans. Theranos had something of that vision with a drop of blood in pharmacies which could also provide very early detection of cancer...but they didn't execute technically among other issues.
I've been looking at this space for a few years. I've come to the conclusion that one of the biggest obstacles to ML replacing/supplementing radiologists is the malpractice ambiguity. If a doctor misses a lesion on a chest Xray, the lawyers know who to sue. What if a ML algorithm that no human understands misses this. Who is liable? It's another sad example of technological progress in healthcare being stifled by bureaucracy / liability concerns.
Liability is of a radiologist is ultimately offset by malpractice insurance. Similarly, the lawyers would go for ML algorithm developers. Now I can see ML insurance business with premiums tied to ML clinical performance ... Obviously there has to be a legislation in place for this.
hey I'm the author, sorry for any confusion, I did separate the hold out sets prior to the image augmentation. Sorry for not making that clear in the post.
22 comments
[ 3.7 ms ] story [ 77.5 ms ] threadI've worked in labs that do this exact same thing. It's cool that they're putting this in a consumer-facing format but the technology (re: accuracy in ML models) still isn't there yet, and there are a lot more performant and sophisticated models than the relatively simple network architectures the authors describe. At risk of being overly critical of the author, to me this just seems like a first, rather naive implementation of ML to be shown as a proof of being able to do ML related things, with the actual performance or usefulness of the model being secondary to that.
In short, putting this into practice at a consumer level: good. Actual modelling: maybe not so good.
> I've worked in labs that do this exact same thing. It's cool that they're putting this in a consumer-facing format but the technology (re: accuracy in ML models) still isn't there yet
Don't these observations explain each other? The whole point of have a consumer product is to reduce office visits and/or increase the monitoring frequency. So reduced accuracy is acceptable since any positive results would be followed up by the standard, more sophisticated tests.
They aren't trying to mimic the state of the art in a phone.
False negative (24%) means you are not getting the treatment you need as soon as you could. Imagine people delaying a real mammogram for a few years because they have an app that says "all good!".
False positive (35%) means unneeded tests and doctors visits. This is less concerning as a bad outcome, but it does mean more stress , more expense and longer wait times to schedule a doctor visit.
This is an interesting exercise in ML, but there is little chance this app would get approval as a diagnostic tool.
Also, FTA:
> The code will be the deliverable to iSono Health as a baseline model for further algorithm development.
I note they didn't give a ROC curve or d'.
Honestly the more interesting case is probably this + mammo, as there is research suggesting sensitivity increases with US + mammo, over mammo alone.
So it's worth looking at.
What they have now is worse than useless, though.
Because the data augmentation is performed prior to the random train/val/test split, nearly identical instances may be found in each set. In other words, a 1 degree rotation of training image X may be found in the validation or test set which would artificially inflate performance.
A game-changer in cancer treatment would be "early detection". Once it gets to be inexpensive to analyze images, then we can have vastly earlier cancer detection...you could work into a scanning machine in a local pharmacy, get scanned & get the analysis right there...you could go in for monthly scans. Theranos had something of that vision with a drop of blood in pharmacies which could also provide very early detection of cancer...but they didn't execute technically among other issues.
https://www.google.co.nz/amp/s/www.washingtonpost.com/amphtm...