IBM is doing the smoke and mirros and selling bullshit game for a while now.
And it's intentional! They have a strategy of inventing new words that sound all scientific (like 'cognitive computing') but there is absolute no meaningfull research from them. Its like the whole company has become a parody.
Any companies that allows their employees to talk to IBM has the problem that afterwards their employees understand less of actual problem domain -- and the knowledge they think they gained from that interaction .. its all 'IBM and consultancy buzzwords' rather than the industry-standard terminology fueled by actual research and science.
IBM has some OK offerings but they've really screwed themselves with all this hyped up marketing around Watson. They've become somewhat of a joke within the real data science community and business executives are also increasingly becoming disillusioned when projects based on IBMs mystery black boxes fail to deliver.
The MD Anderson situation was a debacle of monumental proportions. Such claims on cancer care are quite unethical in my view given that it gave a lot of very sick people false hopes and diverted a lot of hospital attention and funding away from stuff that actually works.
We have a lot of IBM tools at my company, and they're all terrible. I always heard these great things about Watson. Thank you for setting this straight. My world view is healed.
Lots of IBM where I work too, can't say a single good thing either.. honestly Watson being a bust is not very surprising! The people who pay the bills get taken to fancy parties and dinners by IBM, they give us free "trainings"(sales pitches) where they try to convince us to use something that management has already bought into... Honestly it's all very terrible :(
My sentiment of IBM these days is they sell software which is purposely made bad...and people buy it because, well no one ever got fired for going with IBM.
IBM workflow was surprisingly well thought out. You could model fluid flows through humans and services. Had a good deployment model I that the flows were scripts unlike IBM WPS 7 which had them embedded in EARs.
The pattern I've seen: IBM sells things that underdeliver, the solution is always to buy more IBM to fix the inadequacies of whatever it is you've bought already. The go-to line is "Oh that's a know problem with X 3.4, if you upgrade to X 4.5 with Y 2.7 that will fix that problem". This process never stops.
Which is partly due to their actual problem solving strategy. Instead of fix the problem with X 3.4, they bought the company that makes Y so they could use that solution. But Y has it's own problems of course. And even if it didn't X should have just been developed along with something like Y. Though it's unlikely that X was even made by IBM anyways.
What's been crazy to me has been discussions with dudes I know who work there telling me that Watson branding has encroached on Cognos offerings and even disrupted sales.
TLDR: Watson has been trained for 6 years to build a knowledge base about 7 types of cancer; therapy recommendations are entered by experts from one top US hospital.
Looks to me like a giant pattern recognition system. The best assessment I found in the article:
> He said later that the background information Watson provided, including medical journal articles, was helpful, giving him more confidence that using a specific chemotherapy was a sound idea. But the system did not directly help him make that decision, nor did it tell him anything he didn’t already know.
>Looks to me like a giant pattern recognition system.
Pretty much all attempts at AI are pattern recognition systems. I think the AI hype will die down a bit, after we see more of the high profile AI programs underdelivering on marketing's promises. But we'll get some gems that do deliver something useful.
I just guess IBM bit off more than they can chew on this one. It's an old, slow, uninnovative company, trying to do what the cool kids are doing by putting all of their money on it.
> Pretty much all attempts at AI are pattern recognition systems
Pattern recognition is a fundamental component of intelligence, but not all attempts at AI are based on supervised learning (ie "simple" pattern recognition).
Yay, and now we're back at expert systems again. If we had known the AI winter would have been the dark ages and we'd have to reboot civilization again maybe we would have kept some fires on for warmth.
> On a recent morning, the results for a 73-year-old lung cancer patient were underwhelming: Watson recommended a chemotherapy regimen the oncologists had already flagged.
> “It’s fine,” Dr. Sujal Shah, a medical oncologist, said of Watson’s treatment suggestion while discussing the case with colleagues.
> He said later that the background information Watson provided, including medical journal articles, was helpful, giving him more confidence that using a specific chemotherapy was a sound idea. But the system did not directly help him make that decision, nor did it tell him anything he didn’t already know.
I don't know, that doesn't sound bad at all. Perhaps it's just a case of overhyping. But if a computer can do recommend the same treatment as an expensive oncologist with years of training that's actually pretty good!
On the other hand it is quite baffling why doesn't FDA require a clinical trial or any independent review of its capabilities before allowing its use in clinic. It's a bit like Boeing certifying its own planes, I guess.
It's definitely part of medical treatment. Laws are changing to include "software medical devices" in health regulation. The FDA recently recognized this as a gap.
BTW I've worked with regulated software and it's not pretty. Just to update the firmware you have to fill out a ton of paperwork (because somebody died once after a firmware update and the firmware update was found to be the root cause). Regulation and review doesn't really make things tremendously safer, it just provides an audit chain so you can point at something when it does go pear-shape
I can't tell if you're arguing that it is currently regulated, that it will be, or that it fits some platonic notion of 'medical treatment' and therefore should be (even if it's not beneficial).
Laws are changing to include "software medical devices" in health regulation. The FDA recently recognized this as a gap.
This isn't quite correct - FDA has always regulated software medical devices. It is true however that the FDA is looking a lot closer at software than they did in previous decades, and is releasing guidance (e.g. recently on cybersecurity) that signals intentions moving forward. They definitely have signaled that they don't believe they were focused enough on software.
It's important understand that FDA approval to market is always done in terms of indications for use. You don't approve a device full stop, you approve it (and evaluate it) in terms of the indications. Only in this context can you determine if you have a 510k path (i.e. short cut) or need a PMA (i.e. clinical trial). It is also important to realize that different panels in the FDA (i.e. the type of device you are making) have different evaluation criteria.
So "Watson" is unlikely to ever be cleared per se (it is a tech platform) but you are likely to see individual Watson enabled systems file 510ks for particular indications. If I recall correctly this has already happened.
Do you remember what it's already happened for? Do you know what devices they're equivalent to?
While I get the impression FDA and industry prefer 510ks as much as possible, personally I don't see how this is close enough to anything out there right now for that to pass the smell test.
My guess would be that some `Watson` equivalent stuff will be 510k'd in as part of an MRI, CT or DBT imaging system or a CAD tool working on these systems. I think it'd take a while though, I don't think the regulatory science is there yet for actually showing value.
I've seen my doctor consulting Wikipedia before, along with other online resources. I don't think the FDA regulates that type of thing because it's just information, which the doctor is free to use or discard. I would think Watson or any other kind of medical diagnosis system would fall into the same category, as long as the human doctor is making the final decisions.
> Hospitals pay a per-patient fee for Watson for Oncology and other products enabled by the supercomputer. The amount depends on the number of products a hospital buys, and ranges between $200 and $1,000 per patient.
Sigh. Let's remember that in medicine (a severely resource-constrained system), costs operate in a zero-sum environment. $200-$1,000 spent on Watson is $200-$1,000 not spent on some other treatment or human labor. And, in our current environment, specialist doctors will always have to re-do the diagnosis suggested by Watson, because guess who is held legally responsible, in the case of a malpractice suit? Not IBM, but the doctor co-signing the "recommendation".
It's interesting that Watson appears to rely so much on processing text, which runs into so many difficulties. It would be helpful if (for a given disease) every patient's medical records were tracked as structured data. Machine learning would then be able to find patterns of diagnosis much more easily.
(I guess companies like Flatiron Health are working in that direction.)
People have been working on this for 30 years, and the mess keeps getting worse. A significant amount of that mess has been created by engineers trying to solve the wrong problems, or the wrong parts of them.
> “It’s been a struggle to update, I’ll be honest,” said Dr. Mark Kris, Memorial Sloan Kettering’s lead Watson trainer. He noted that treatment guidelines for every metastatic lung cancer patient worldwide recently changed in the course of one week after a research presentation at a cancer conference. “Changing the system of cognitive computing doesn’t turn around on a dime like that,” he said. “You have to put in the literature, you have to put in cases.”
Wow the way they describe Watson is just like a massive book of documentation, with some NLP functionality layered on top to interface with the doctors. Keeping the database up to date must be a huge undertaking. Also this:
> "Given the same clinical scenario, doctors can — and often do — disagree about the best course of action, whether to recommend surgery or chemotherapy, or another treatment. Those discrepancies are especially wide for second- and third-line treatments given after an initial therapy fails, where evidence of benefits is slimmer and consensus more elusive."
Having an expert doctors opinion hard coded into the machine sounds good, but it won't be as good as a doctor in the room who can pick up on all sorts of other cues that I'm not sure how'd you even type into a computer. Things like breathing patterns, reaction to tactile stimuli, fuzzy logic ('that hurts, that hurts more'), facial expressions and so on.
Seems like this project has a long way to go, to get to the standard the IBM marketing team are pretending it's already at.
"He noted that treatment guidelines for every metastatic lung cancer patient worldwide recently changed in the course of one week after a research presentation at a cancer conference."
Isn't this a good thing then that there's a digital assistant to ensure doctors are up-to-date and provide the right treatment plans?
Of course, but in the very next sentence the doctor says the system stays out of date for quite a while. That's probably because they need an engineer to find all the old information and remove it, then input all the new information in whatever standard Watson requires.
A digital assistant that signals guideline changes does not require advanced technology. This reflects failure of the medical system, more than anything else.
Plus, the fact that guidelines change suddenly after some conference should tell you that treatment in the case depicted here is defined entirely on expert opinion. AI should seek to do better than just following expert opinion.
Conclusion: the Watson system is a failure.
Cynical take, having worked in organisations that have bought into IBM: IBM is not into curing cancer, it's into selling, and in particular up-selling at enormous scales (government and big business). The primary goal of the Watson project has not been to cure cancer, or do anything in particular, but to get existing customers to buy into extremely expensive contracts that include 'Watson' in some way. All the hype, which it is now becoming apparent was artificially expanded to some extent, is just a means to this end. As with most (all?) of their other products, it'll be oversold, underdeliver, but lock organisations into even heftier contractual chains.
As per every thread Watson is brought up in, it's mainly a catch-all term for its consulting division. IBM's consultants aren't very good, and the good ones move on to brighter futures.
This is why Watson is a disappointment and every Watson project ends in cost overruns and failure.
There's a bigger issue at hand here. AI generally has not made huge inroads into disease treatment. I think this is because we haven't figured out a good way to describe the diseases. Where it does work, we have a good way to classify the data (i.e. a set of training images or clear metrics to feed it). As Google/Verily learned the hard way, its not easy to fit digital models to organic beings.
In the area of cancer treatment, my guess is that the reason specialists are highly paid/trained is because the analysis and judgement required is highly manual and varied. Surgical intervention, prescription medications, targeted radiation treatment machines, patient psychology and counseling. Other than maybe guessing the prescription based on symptom list, what could the AI profitably help with?
Its kind of sad to see what has become of IBM. It is now just another consulting shop that wants to plant a slew of 3-400/hr consultants to run projects. The big consulting companies best trick was to convince those in the enterprise that hiring a boutique consulting shop is "risky" and that a big 4/5 consulting company is a "safe" option even though it is orders of magnitude more expensive and not necessarily better, and in some cases worse when it comes to specialized systems.
Source: years of working in Energy/Commodity trading systems market and seeing several botched big 4 implementations that vendor/boutique shops can do at 1/10th the price.
Having done big and small consulting work, I think part of the issue is that waterfall still has quite a large buy-in. Smaller shops tend to go more agile, but when you have a team of analysts chomping at the bit to document your processes and say "yes, we can do this" (regardless of if they actually can) that builds confidence.
And that confidence is what moves the project forward, until the customer sees dates slipping, at which point it's too late.
> On a recent morning, the results for a 73-year-old lung cancer patient were underwhelming: Watson recommended a chemotherapy regimen the oncologists had already flagged.
> He said later that the background information Watson provided, including medical journal articles, was helpful, giving him more confidence that using a specific chemotherapy was a sound idea. But the system did not directly help him make that decision, nor did it tell him anything he didn’t already know.
This type of thinking is reason true ML adoption is still sparse. Analytics should be viewed as a supplement to decision making, not a replacement.
Manipulative marketing tactics by IBM and others are largely to blame. I'm confident that over time, as the general population becomes better educated on the topic, the perspective will shift.
54 comments
[ 3.3 ms ] story [ 49.3 ms ] threadbut on the other hand, IBM's negotiation with MD anderson was one kind of a deal - instead of paying for hospital data, IBM made hospital pay...
And it's intentional! They have a strategy of inventing new words that sound all scientific (like 'cognitive computing') but there is absolute no meaningfull research from them. Its like the whole company has become a parody.
Any companies that allows their employees to talk to IBM has the problem that afterwards their employees understand less of actual problem domain -- and the knowledge they think they gained from that interaction .. its all 'IBM and consultancy buzzwords' rather than the industry-standard terminology fueled by actual research and science.
The MD Anderson situation was a debacle of monumental proportions. Such claims on cancer care are quite unethical in my view given that it gave a lot of very sick people false hopes and diverted a lot of hospital attention and funding away from stuff that actually works.
My sentiment of IBM these days is they sell software which is purposely made bad...and people buy it because, well no one ever got fired for going with IBM.
Looks to me like a giant pattern recognition system. The best assessment I found in the article:
> He said later that the background information Watson provided, including medical journal articles, was helpful, giving him more confidence that using a specific chemotherapy was a sound idea. But the system did not directly help him make that decision, nor did it tell him anything he didn’t already know.
Pretty much all attempts at AI are pattern recognition systems. I think the AI hype will die down a bit, after we see more of the high profile AI programs underdelivering on marketing's promises. But we'll get some gems that do deliver something useful.
I just guess IBM bit off more than they can chew on this one. It's an old, slow, uninnovative company, trying to do what the cool kids are doing by putting all of their money on it.
All attempts in modern AI. Most interesting research from the 60s and 80s wasn't based on pattern recognition.
Pattern recognition is a fundamental component of intelligence, but not all attempts at AI are based on supervised learning (ie "simple" pattern recognition).
> “It’s fine,” Dr. Sujal Shah, a medical oncologist, said of Watson’s treatment suggestion while discussing the case with colleagues.
> He said later that the background information Watson provided, including medical journal articles, was helpful, giving him more confidence that using a specific chemotherapy was a sound idea. But the system did not directly help him make that decision, nor did it tell him anything he didn’t already know.
I don't know, that doesn't sound bad at all. Perhaps it's just a case of overhyping. But if a computer can do recommend the same treatment as an expensive oncologist with years of training that's actually pretty good!
On the other hand it is quite baffling why doesn't FDA require a clinical trial or any independent review of its capabilities before allowing its use in clinic. It's a bit like Boeing certifying its own planes, I guess.
Because it's a tool for informing doctors, not a medical treatment. We don't have a clinical trial before PubMed rolls out a new search form either.
BTW I've worked with regulated software and it's not pretty. Just to update the firmware you have to fill out a ton of paperwork (because somebody died once after a firmware update and the firmware update was found to be the root cause). Regulation and review doesn't really make things tremendously safer, it just provides an audit chain so you can point at something when it does go pear-shape
It's important understand that FDA approval to market is always done in terms of indications for use. You don't approve a device full stop, you approve it (and evaluate it) in terms of the indications. Only in this context can you determine if you have a 510k path (i.e. short cut) or need a PMA (i.e. clinical trial). It is also important to realize that different panels in the FDA (i.e. the type of device you are making) have different evaluation criteria.
So "Watson" is unlikely to ever be cleared per se (it is a tech platform) but you are likely to see individual Watson enabled systems file 510ks for particular indications. If I recall correctly this has already happened.
While I get the impression FDA and industry prefer 510ks as much as possible, personally I don't see how this is close enough to anything out there right now for that to pass the smell test.
My guess would be that some `Watson` equivalent stuff will be 510k'd in as part of an MRI, CT or DBT imaging system or a CAD tool working on these systems. I think it'd take a while though, I don't think the regulatory science is there yet for actually showing value.
There have been approved CAD systems separate from imaging systems since the 90s.
"Value" is a nebulous term. FDA looks specifically for "safety & efficacy", and the criterion for each are rather dependent on the intended use.
You're not buying thousands consulting hours and some big data services... you're buying Watson
0: https://www-01.ibm.com/software/passportadvantage/pvu_licens...
Sigh. Let's remember that in medicine (a severely resource-constrained system), costs operate in a zero-sum environment. $200-$1,000 spent on Watson is $200-$1,000 not spent on some other treatment or human labor. And, in our current environment, specialist doctors will always have to re-do the diagnosis suggested by Watson, because guess who is held legally responsible, in the case of a malpractice suit? Not IBM, but the doctor co-signing the "recommendation".
(I guess companies like Flatiron Health are working in that direction.)
Wow the way they describe Watson is just like a massive book of documentation, with some NLP functionality layered on top to interface with the doctors. Keeping the database up to date must be a huge undertaking. Also this:
> "Given the same clinical scenario, doctors can — and often do — disagree about the best course of action, whether to recommend surgery or chemotherapy, or another treatment. Those discrepancies are especially wide for second- and third-line treatments given after an initial therapy fails, where evidence of benefits is slimmer and consensus more elusive."
Having an expert doctors opinion hard coded into the machine sounds good, but it won't be as good as a doctor in the room who can pick up on all sorts of other cues that I'm not sure how'd you even type into a computer. Things like breathing patterns, reaction to tactile stimuli, fuzzy logic ('that hurts, that hurts more'), facial expressions and so on.
Seems like this project has a long way to go, to get to the standard the IBM marketing team are pretending it's already at.
Isn't this a good thing then that there's a digital assistant to ensure doctors are up-to-date and provide the right treatment plans?
http://www.underconsideration.com/brandnew/archives/new_logo...
This is why Watson is a disappointment and every Watson project ends in cost overruns and failure.
In the area of cancer treatment, my guess is that the reason specialists are highly paid/trained is because the analysis and judgement required is highly manual and varied. Surgical intervention, prescription medications, targeted radiation treatment machines, patient psychology and counseling. Other than maybe guessing the prescription based on symptom list, what could the AI profitably help with?
Perhaps more likely, human doctors are actually quite good and AI has yet to out do them.
Source: years of working in Energy/Commodity trading systems market and seeing several botched big 4 implementations that vendor/boutique shops can do at 1/10th the price.
And that confidence is what moves the project forward, until the customer sees dates slipping, at which point it's too late.
> He said later that the background information Watson provided, including medical journal articles, was helpful, giving him more confidence that using a specific chemotherapy was a sound idea. But the system did not directly help him make that decision, nor did it tell him anything he didn’t already know.
This type of thinking is reason true ML adoption is still sparse. Analytics should be viewed as a supplement to decision making, not a replacement.
Manipulative marketing tactics by IBM and others are largely to blame. I'm confident that over time, as the general population becomes better educated on the topic, the perspective will shift.