> Admittedly though given their pretty large size I can't name much else.
Exactly. Even if 20,000 people worked on those supercomputers, mainframes and Watson, what do the other 350,000 employees work on? Consulting, and it's been that way since Gerstner.
I was going to agree with you, but then my last name is Watson :-)
Seriously, I used IBM Watson on a consulting project a little over two years ago and I was disappointed. To be fair I should take another look. IBM has bought some good companies whose work is exposed in BlueBiz and other web services and the permanent free tier levels they give away rival the free tier levels from GCP.
I think AutoML and auto data science web hosted offerings will be a huge market with right now Google, Amazon, and Microsoft leading the way. If I worked at IBM in a position of power, I would work hard to create a great developer experience for simplifying the use of machine learning, NLP, etc.
IBM bit off more than they could chew with the AI/NLP project and their brand suffered as a consequence. Some problems are simply hard in the general case, though there are some spin-outs and spin-offs of that project for more limited domains that are successful.
Comparing the amount their brand suffered vs profits they may have come out ahead. AI got some real products from this hype cycle, but expectations where high enough that a crash was inevitable.
The big difference with Google and AutoML is that Google was already doing lots of machine learning in the past, just not deep learning.
With deep learning it took the transition seriously internally as well: there's a huge push to get all models to use Tensorflow - even if the transition itself is painful - and this helps the infrastructure providers as well to gather feedback.
With IBM it all started as a very successful research project, but instead of working more on the Watson system, it decided to rename dumber projects to be under the Watson umbrella as well thereby giving the impression that it's the same system that won in Jeopardy.
I used some Watson stuff that came with another project as well.
It wasn’t bad at all, we didn’t cure cancer but it provided a not so awful way to do some basic ML workloads. The other problem is that IBM pitches Watson to anyone, and many of the customers lack the clue to utilize it.
Not sure about Watson, but there was just this week a story about using pattern matching to improve breast cancer screening imaging. It seems the tech is decent but the problem is, it's unclear whom to sue if needed, so there's an obstacle to using it.
The fact Bitcoin is worth $90 billion and has survived a decade of endless criticism, means blockchain is successful beyond anyone's wildest dreams a decade ago. I would say Bitcoin and Ethereum have been way more successful than Watson has.
People kept investing with Bernie Madoff for decades too, because his investments kept giving returns. People will keep shoveling money into Bitcoin as long as its value keeps swinging wildly. That's hardly a measure of success.
How much will Bitcoin have to be worth, and for how long, to make it successful in your eyes? Or is the entire idea of a digital asset with intrinsic value impossible to you?
USDC (https://www.circle.com/en/usdc) is a dollar-backed cryptocurrency operating on Ethereum. Over $260 million has been converted into USDC, and it is used all over the globe to move small or very large amounts of money securely within seconds (https://etherscan.io/token/0xa0b86991c6218b36c1d19d4a2e9eb0c...). This is a novel application of blockchain technology that is beyond a slot machine.
The thing I think is odd about Bitcoin and it's future is how energy intensive it is to run the network. It is known that bitcoin miners use huge amounts of electricity, which costs money, but you can't yet pay your electric bill with bitcoin, nor can you pay taxes in bitcoin. Just seems a bit unsustainable to me.
The hype around blockchain is far more than just cryptocurrencies, which have done well by many measures.
The person you're replying to is saying that all the hype around blockchain for contracts, identifiers etc. is most likely way overblown and we will look back in a few years and see it as a mostly useless fad
The Federal Reserve is worth maybe every dollar on the planet. It's survived almost a hundred years. It bankrolled Bitcoin with it and most cryptocurrencies depending on dollars for investments, paying employees, etc. From centralized banking to sustaining decentralized platforms, the Federal Reserve is successful beyond anyone's wildest dreams in the 1910's. Except maybe the people that created it.
See why your argument doesn't prove Bitcoin is a good thing in the long term and/or if you're a believer in its superiority to centralized banking?
I made no comment on Bitcoin or other cryptocurrencies replacing the Federal Reserve. I am arguing that Bitcoin and other cryptocurrencies are, 1) Not worthless, 2) Not a scam, 3) Have interesting technical properties that are worthy of study and improvement, 4) Have real-world utility, and 5) Should not be immediately dismissed by HN groupthink.
What actually is so hard about AI in health care? Why not just take a set of diagnostic indicators for inputs, map to conditions/treatments as outputs and train a neural net?
Either the AI can generalize or it can't. If it can't, you don't need neural nets at all. If it can, you need to worry about whether it's generalizing correctly, which is definitely "so hard".
Sparse data. Neural nets (for example) do horribly at this sort of thing; they're data hungry.
It's super difficult to beat something like linear regression in this sort of thing (ideally combined with domain knowledge -something neural approaches mostly fail at), and even linear regression gives awful results.
There are huge barriers to getting a healthcare AI product to market in the United States. First off, data is generally extremely difficult and expensive to gather due to the need to protect patient info according to the Health Insurance Portability and Accountability Act (HIPAA). To get representative data, you need to gather from many sites and the data needs to be anonymized and "truthed" by a medical expert who typically must examine multiple medical reports and case history for each patient. Each data collection site requires approval for data collection by an Institutional Review Board, which may take months or even years to get. Assuming you are able to get a reasonable amount of data for development and testing, you need to solve the actual AI problem (probably very difficult due to the extremely heterogeneous nature of disease you are dealing with) and then you need regulatory approval. Depending on the FDA classification of your product, this can be another many-month to many-year process of data collection and clinical studies.
The problem is that the numerous easy cases do not result in a useful network ... any 1st-year intern will get the easy results already.
The hard cases, which would be useful to a doctor, occur very rarely. My father's unusual reaction to a post-bypass drug regimen was something like the 3rd time that happened in Canada. How do you "train" that into a neural network?
Sounds like anomaly detection would be quite useful for monitoring the expected reactions and comparing with actual outcomes.
It may not be the answer to "what is the right drug?" but it may be enough to say "what he got ain't right for him"
I work on an ML project on epilepsy, and this happened all the time. Additionally what the clinicians saw as easy/boring wasn’t always te case for the model.
In addition to what other commenters have written, you also have the problem that your inputs aren't clean or straightforward. People don't always answer fully truthfully, and sometimes give extraneous information that's not germane to their particular complaint. So, even if you get past the regulatory hurdles of obtaining and using data, and you have enough data to train your neural network, and you've handled the huge class imbalance issue with rare diseases, you still have to deal with the fact that the inputs to your model are subject to all kinds of bias and confounding that's associated with human-generated data.
To make this concrete: I used to do a lot of HIV testing and counseling. Whenever I'd ask someone "how often do you use condoms?" they'd 100% of the time say "every time!" When I'd then switch to asking "when was the last time you didn't use a condom?" they'd often reply along the lines of "last week."
This kind of issue happens not just with awkward questions, but also with more "objective" data like labs and medications.
- Was the antibiotic script written for a patient necessary for their condition, or was the physician tired and couldn't fend off a particularly assertive patient who was convinced antibiotics would solve their viral infection?
- Are labs randomly ordered, or ordered in a targeted fashion based on the "hunch" of a physician? If the latter, then the presence of lab result would itself be correlated with having some disease, and could thus throw off your entire model.
- How does the patient's insurance status influence the tests they have ordered? At one clinic I'm aware of, they typically order HIV/GC/Chlam screenings as a bundle, but if the patient has a PPO insurance they are also more likely to throw in a syphilis screening.
- How does patient preference factor into what gets diagnosed, ordered, and treated? You might have two early stage, equal risk breast cancer patients, one whom opts for a double mastectomy because she saw what happened to a coworker who had breast cancer and doesn't want to risk recurrences, whereas the other opts for a lumpectomy because she thinks her risks are low enough to be more conservative in treatment.
These issues are all solvable, but for AI to have a useful impact in healthcare, there's a lot more work than just throwing a bunch of data into a deep learning model.
There are hundreds of thousands of diagnostic and treatment codes, and most patients will have several, or even dozens of codes for a presentation. Add in the patients age and other demographic variables, medical history, and family history, and you have more combinations of variables than patients.
Except for some really common problems, like cases of influenza, every patient is unique.
AI can be used to handle simpler problems - identifying patients at risk of repeat admissions, or flagging the likelihood of dehydration for example.
>”But Watson won’t change its conclusions based on just four patients. To solve this problem, the Sloan Kettering experts created “synthetic cases” that Watson could learn from, essentially make-believe patients with certain demographic profiles and cancer characteristics.”
Is this standard practice in machine learning? This sounds more like regular programming to get exactly the outcome you want.
It's standard practice for idiots. And apparently "Sloan Kettering experts."
There are semi-supervised techniques to do stuff like this in a more systematic/automated way, but you still don't get anything for free: the outcome depends on the priors used to do the semi-supervised voodoo. In a generous moment I might assume this is what they meant, but it's still dumb.
To me, the main mistake was the series of commercials giving the strong impression that IBM already had this incredible Artificial General Intelligence that was indistinguishable from a highly intelligent human and was solving a myriad of difficult practical problems better than any expert. I suspect that most who were well-versed in AI felt the ads were disingenuous from the start. I know I did. I think the marketing campaign would have better served IBM had it laid out their commitment to achieve these things without sounding like they were already there.
CUDA delivered a 10x to 100x performance boost on legacy code for real. The hardware had been capable of doing this for at least a generations previously but it lacked a few operations to make it accessible and efficient.
What I think made people miss it is no one wanted to refactor their code to run efficiently on GPUs. It wasn't that hard to do so in a lot of cases but it was work.
And all this set the stage for deep learning. It was the opening act if you will...
> CUDA delivered a 10x to 100x performance boost on legacy code
CUDA offered a way to access a limited set of operations on special-purpose hardware with a rewrite of critical sections. A pretty far cry from 'boosting legacy code.'
To me it seemed obviously overhyped. Having worked in the machine learning/AI field back in the 90s all the claimed capabilities of Watson didn't seem credible.
IBM have been doing this since Deep Blue, if not older. It's their core business model: build cool tech demos and then sell consulting deployments that use nonexistent features of the tech
I think it runs deeper than that. I think the mythology of the "Electronic Brain" goes back to the 60's and all the Space Age idealism about technology.
Maybe I'm just naive, but I really think that strand still exists in IBM's corporate DNA. They still see this as solvable given sufficient time and effort.
The commercials are the public-facing tip of a much larger iceburg of enterprise-scale sales and marketing. Big institutions gave Watson a shot because they created a machine that maximized FOMO with intriguing demos, reduced risk with the "nobody gets fired for buying IBM effect", and perhaps most importantly pitches projects that were large enough to move the needle and make the career of a high level administrator.
When you realize how much positioning is required for large scale sales, you can start to understand why the same company might not have the feedback loops in place to build a product.
>To me, the main mistake was the series of commercials giving the strong impression that IBM already had this incredible Artificial General Intelligence
100% correct. Had IBM marketed Watson for what it actually is - an enterprise version of Amazon Alexa or Google Assistant (which admittedly didn't even exist at the time) - they'd be in a completely different spot. Instead we got the Bob Dylan commercials (which under the covers was just speech to text, a hardcoded dialog tree, and text to speech) and all the "Watson is curing cancer" nonsense (which under the covers was just a rules engine sitting on top of a search engine intended to provide customized treatment plans). The actual technology works, but it's an NLP platform for building chatbots and search engines (think intent classifier, relevancy ranker, named entity/semantic relationship/sentiment annotators) not the AGI that it was marketed as.
I worked on this project and there were a lot of issues. Two of the biggest were:
* Whatever the quality of the technology (which I personally never saw as that compelling) was wrapped up in terribly written research code, making it practically impossible to setup and use.
* The Jeopardy demo was made possible by the existence of a marked-up source of general knowledge (Wikipedia), a ready-made bank of questions and answers from past shows (j-archive.org), and the fact that practically anyone had the ability to curate more Q&A pairs. This is almost totally different than the medical use case where the knowledge is wrapped up in proprietary textbooks and papers and the only people able to curate training data are medical professionals.
That's not really true. The UMLS has a large graph of marked up medical domain knowledge that can be used. It's not as specific as one might want for developing an AI autodoc, but it's quite a bit better than what is available in most fields. It's actually quite similar to what one can derive from Wikipedia.
Unfortunately UMLS and Wikipedia are not good enough for drug information. Structuring that data requires a fine balance of tooling (with some NLP) and a good team of biocurators (doctors, pharmacists, pharmacologists and domain experts).
Skip to Jae Won Joe's answer to see a case study with a patient showing the interactive process. Then, especially look at ending questions about missing or deceptive data. Seems like their good diagnoses comes from a combination of domain data and expertly reading people in front of them. Machines suck at the part I emphasized. The data sets might not reflect a lot of stuff like that, too. Who knows.
This is a great run-down of the realizations that a lot of the engineering team came to and what I was trying to touch on in my reply above. There is just so much information & context within a doctor's head and so much that is going on during an examination that looks simple to our eyes. Even in the idealized case of having accurate, structured patient data and focusing only on a simple disease, it's a challenge to diagnose and prescribe a correct treatment. Even a system that can accept the text of a patient's medical record and successfully pull out relevant details would be challenge.
I think this article addresses the half the medical landscape where the problems are hard and computers are worse than humans at solving them. The other half is where the problems are "easy". Practices where Doctors spend ~5 minutes with each patient doing physicals and treating minor ailments. In that half doctors can already easily handle the problems so there's no real value add for a computer.
It would take a paradigm shift before AI became useful. Going to watson.com and getting a prescription for antibiotics would be useful and technically feasible. So would augmenting lower trained people to be able to deliver more care. Neither are possible legally and it's not exactly a field you can get away with disregarding the law.
Having that taxonomy is definitely a great starting point for the field, but just to build a model to extract patient data from the text of a medical record you'd need to:
* Procure patient medical files. This rabbit hole includes things like HIPPA compliance and sufficiently anonymizing the files. Also, hospitals consider 3 years of information on 100 patients to be a lot of data.
* Have trained medical professionals annotate the files. Good luck convincing doctors to do this. Also good luck in determining what to do if the handful of doctors you're granted a few hours with a week can't agree on annotations.
* Actually produce a model that is reasonably good at pulling information out of patient charts,
Even after all the above, you still only have a solution that is giving you structured data, which is something that other EMRs can provide w/o the problems of accuracy & completeness of NLP process. There is still the process of diagnosis and treatment and also the fact that this is all a loop ('active learning').
A bunch of the projects described, and the (technical) difficulties encountered, make me wonder if GPT-2-style systems would have better odds. Is anyone looking into applying that to medical/scientific text NLP problems?
I mean, GPT-2 still often produces nonsense more similar to dream imagery than useful reasoning, but I gather most medical residency students are half-asleep most of the time anyway, so... :)
I was recently diagnosed with neuroendocrine cancer, which my PCP had been misdiagnosed for 10 years as IBS. This is more the norm than the exception for people with this type of cancer, Steve Jobs included. It's a perfect example of where AI can likely diagnosis what my PCP could not. AI tools need to find cancer problems to solve which are more suited to their capabilities. e.g. does anybody know of a company working on "AI for cancer screening"? This is desperately needed and would have helped me.
I'm am sorry to hear that. I don't know about companies, but cancer screening with machine learning is a very active topic in academia at least. Other topics include outcome prediction, and analysis of treatment alternatives.
Thanks. Actually, my cancer is very weak. On a scale of 1 to 10, it's aggressiveness is 1-2. But one of the liver tumors is very and will be removed soon. I am starting a side project to work on cancer screen if anyone can point me to anyone willing to partner on this, I would appreciate the help.
Unfortunately, there are still many cancers detected far too late for effective treatment. It sounds like you were indeed fortunate to have a less aggressive form. AI for cancer screening generally falls under the category of "Computer-Aided Detection" or CAD. The commercial and academic CAD efforts tend to be organized by the primary anatomical site of cancer and the detection method (e.g. X-Ray, CT-Scan, PET, ultrasound, blood test) . Was your primary the pancreas or intestine? Are you wanting to contribute to an imaging detection method or something else? I might be able to help you identify someone working in the area depending on your goals.
My cancer was finally found in my intestine. This is another opportunity. My primary oncologist and local surgeon were telling me the primary tumor, which has metastasized to a very large liver tumor, could not be found. I did my own research and found they had ordered the wrong type of imaging scan. Only after I pushed to have the correct scan (Gallium 68 PET/CT scan) was the primary tumor found. This was a "lack of information" for my local oncologist. Computer-aided diagnosis would have helped him. An additional new symptom (flushing) appeared and my PCP recognized a specific cheap blood test was needed that led to the cancer being found. I am happy to contribute to an imaging study. But I want to work on cancer screening. What kind of automation/screening would be needed to prevent 10 years of misdiagnosis by my PCP? ... not only for this type of cancer but for all of the top 15-20 types of cancer. People are not being screened. How can we make screening affordable? And how can we raise awareness of possible misdiagnosis and or affordable screening?
There currently is no good candidate for a imaging modality that can be used for a general screening program to find the top 15 to 20 cancers and I am unaware of anything on the near horizon. Such a scan would have to examine the neck thru the groin area to cover even just 10 out of the top 15 or so cancer types. Since screening involves patients with no symptoms, most patients won't actually have any disease and thus the imaging must be inexpensive, must have high sensitivity, must have a reasonable false positive rate, must involve little to no radiation, and must not require injection of contrast agents or radioactive tracers. That eliminates all of the imaging modalities I can think of that can examine large areas of the body for cancer. The best we have today are compromises on these criteria for patients that are at relatively high risk, such as a smoker or a cancer survivor, or for highly focused screening programs such as what we have for breast cancer.
The best reference about Watson last year was on reddit, "Watson is a brand, not a "thing". It covers any technologies that IBM sells in the cognitive space. They are further broken up into the business areas."
All hype and marketing built on open source tools, all while AWS, Google, and Azure build out bigger cloud offerings. As OP mentioned, they used to be a great company.
Watson should rebrand then. You can't have a colossal failure such as the one reported by Stat News: “multiple examples of unsafe and incorrect treatment recommendations.” — [Memorial Sloan Kettering Cancer Center] and expect to move on from it as if they were growing pains.
If you can tune the specifics-sensitivity curve, you should be able to handle both cases. You have to be willing to refuse to provide an answer when you have low confidence.
The existence of adversarial attacks with high confidence on virtually all production ML systems should indicate that confidence numbers are not enough to rely on.
Do these attacks require fine control over the input? Eg. if you are scanning a patient’s body, does it matter if the model can be fooled by editing the values of individual pixels? This implies that you have a threat model where the data coming from the sensor is being manipulated, in which case all bets are off (the image could be entirely replaced). It doesn’t seem much different from a statistical model that you can blow up by feeding in values designed to cause a divide by zero (values that wouldn’t appear in real-world data).
It seems like a problem when classifying user-provided images (eg. identifying obscene images on a social network) but not so relevant when you own the sensors.
You do have a point - all the adversarial attacks on ML models rely on full adversarial control on the inputs, which probably isn't the case with medical records. If there was unconstrained access to a patient's MRI scans by an adversary, then I don't think adversarial attacks on the ML diagnostic models are the biggest problems you'll face.
[from wikipedia] Thomson Reuters sold Thomson Healthcare to Veritas Capital for US$1.25 billion On June 6, 2012. The new company, Truven Health Analytics, became an independent organization solely focused on healthcare. Truven is a portmanteau of the words "trusted" and "proven". IBM Corporation acquired Truven on February 18, 2016, and merged with IBM's Watson Health unit.
The greatest thing about BlueMix was its promotional budget. Saw The Force Awakens on IBM's dime, and got about $50 worth of movie theater gift cards. All I had to do was sit through a 20 minute presentation about BlueMix case study integrations.
I did look at the BlueMix platform, and it seemed like a big mix of various APIs I could tap into. Some of them looked neat, but I never had a reason to really use any of them.
I always find it kind of silly when AI is just thrown into a field with the notion that they'll just deal with the messy, subjective and unstructured data (like hand written medical notes for example) as is. For me it makes much more sense to try to clean up and structure the data from the start instead. Maybe come up with some data acquisition compromise that is both UX friendly and give rise to more structure and consistency.
My company does machine learning checked by physical models. Our single biggest problem (and management is finally waking up to it) is curating the incoming data. And this is in a mature industry (oil & gas).
My biggest surprise has been how little everyone is aware of their data quality.
The only explanation I've been able to come up with is that when it's all human processed, Joe 2nd-link-in-the-chain just deals with all the inconsistencies as best he can to get his job done, and never reports issues up.
But Joe typically hates dealing with the inconsistencies, and he can tell you exactly how they could be fixed.
It generally seems like (a) the suggestions for fixes are impractical to implement (overly detrimental effect on counterparty), (b) Joe isn't empowered organizationally to suggest fixes that will be implemented, or (c) Joe doesn't have access to the IT tools to implement fixes himself.
Main problem is Watson its like a million different things all with the same name. So no one actually knows what they are selling when they talk to you about Watson. Literally I had one IBM guy ask to check out their Watson and it took 5 minutes of them showing me the demo before I figured out what it was (cloud based Nvidia Digits alternative). If that particular Watson had an actual name, I probably would've heard about it before and I'd actually be able to recommend it to someone.
During my time in IBM I was walking by the "main" Watson server everyday into work. I was always impressed at the package, and what ever they said it could do.
Finally got my hands on it and was incredibly disappointed. Lacking features, archaic interface, and not delivering on what it was promised. Talked to some guys working for the Watson team and they told me that half of the AI stuff is just them doing it manually under the guise of "integration and configuration".
I did have fun "talking" to Watson whenever I was bored at work, trying to make him swear
126 comments
[ 4.6 ms ] story [ 199 ms ] threadAnyone surprised by this?
For instance, they developed software for the Apollo mission and the Space Shuttle. The IBM/360. The IBM PC. AS/400.
Do you have a more recent example? something that happened several years after Louis Gerstner first assumed leadership.
Admittedly though given their pretty large size I can't name much else.
Exactly. Even if 20,000 people worked on those supercomputers, mainframes and Watson, what do the other 350,000 employees work on? Consulting, and it's been that way since Gerstner.
I hope that's not the case for their quantum computer
Seriously, I used IBM Watson on a consulting project a little over two years ago and I was disappointed. To be fair I should take another look. IBM has bought some good companies whose work is exposed in BlueBiz and other web services and the permanent free tier levels they give away rival the free tier levels from GCP.
I think AutoML and auto data science web hosted offerings will be a huge market with right now Google, Amazon, and Microsoft leading the way. If I worked at IBM in a position of power, I would work hard to create a great developer experience for simplifying the use of machine learning, NLP, etc.
With deep learning it took the transition seriously internally as well: there's a huge push to get all models to use Tensorflow - even if the transition itself is painful - and this helps the infrastructure providers as well to gather feedback.
With IBM it all started as a very successful research project, but instead of working more on the Watson system, it decided to rename dumber projects to be under the Watson umbrella as well thereby giving the impression that it's the same system that won in Jeopardy.
It wasn’t bad at all, we didn’t cure cancer but it provided a not so awful way to do some basic ML workloads. The other problem is that IBM pitches Watson to anyone, and many of the customers lack the clue to utilize it.
The problem is that Watson was sold as a kind of "AI Doctor", which is pretty much ludicrous
> Overpromised and Underdelivered
Sounds kind of catchy, to be honest.
How much is Bitcoin worth without translating it to another currency? Schrute Bucks and Stanley Nickels.
The person you're replying to is saying that all the hype around blockchain for contracts, identifiers etc. is most likely way overblown and we will look back in a few years and see it as a mostly useless fad
See why your argument doesn't prove Bitcoin is a good thing in the long term and/or if you're a believer in its superiority to centralized banking?
It's super difficult to beat something like linear regression in this sort of thing (ideally combined with domain knowledge -something neural approaches mostly fail at), and even linear regression gives awful results.
The hard cases, which would be useful to a doctor, occur very rarely. My father's unusual reaction to a post-bypass drug regimen was something like the 3rd time that happened in Canada. How do you "train" that into a neural network?
To make this concrete: I used to do a lot of HIV testing and counseling. Whenever I'd ask someone "how often do you use condoms?" they'd 100% of the time say "every time!" When I'd then switch to asking "when was the last time you didn't use a condom?" they'd often reply along the lines of "last week."
This kind of issue happens not just with awkward questions, but also with more "objective" data like labs and medications.
- Was the antibiotic script written for a patient necessary for their condition, or was the physician tired and couldn't fend off a particularly assertive patient who was convinced antibiotics would solve their viral infection?
- Are labs randomly ordered, or ordered in a targeted fashion based on the "hunch" of a physician? If the latter, then the presence of lab result would itself be correlated with having some disease, and could thus throw off your entire model.
- How does the patient's insurance status influence the tests they have ordered? At one clinic I'm aware of, they typically order HIV/GC/Chlam screenings as a bundle, but if the patient has a PPO insurance they are also more likely to throw in a syphilis screening.
- How does patient preference factor into what gets diagnosed, ordered, and treated? You might have two early stage, equal risk breast cancer patients, one whom opts for a double mastectomy because she saw what happened to a coworker who had breast cancer and doesn't want to risk recurrences, whereas the other opts for a lumpectomy because she thinks her risks are low enough to be more conservative in treatment.
These issues are all solvable, but for AI to have a useful impact in healthcare, there's a lot more work than just throwing a bunch of data into a deep learning model.
Except for some really common problems, like cases of influenza, every patient is unique.
AI can be used to handle simpler problems - identifying patients at risk of repeat admissions, or flagging the likelihood of dehydration for example.
Is this standard practice in machine learning? This sounds more like regular programming to get exactly the outcome you want.
There are semi-supervised techniques to do stuff like this in a more systematic/automated way, but you still don't get anything for free: the outcome depends on the priors used to do the semi-supervised voodoo. In a generous moment I might assume this is what they meant, but it's still dumb.
- if you're a generally competent, learned person
- and the new product feels like a giant leap from anything that exists already
- it's likely not all it claims to be.
If anyone is promising a 20% or greater improvement over the current leaders, I immediately file it away as a scam until I see extraordinary proof.
Other than some very immature industries, even 20% leaps don't happen without earth shattering once-in-a-generation breakthroughs.
Some call me a pessimist, and occasionally I'm unfairly dismissive of new technologies. But for every one I get wrong, I get 99 others correct.
Mature and/or competitive.
20%+ without a breakthrough can absolutely happen in niche industries that have historically lacked competition.
What I think made people miss it is no one wanted to refactor their code to run efficiently on GPUs. It wasn't that hard to do so in a lot of cases but it was work.
And all this set the stage for deep learning. It was the opening act if you will...
CUDA offered a way to access a limited set of operations on special-purpose hardware with a rewrite of critical sections. A pretty far cry from 'boosting legacy code.'
Ha, this is just silly. Ok, so what percent were IBM promising? How do you ever reduce anything complicated into a percent change.
Sell consulting deployments that hope to fund development of promised features of the tech.
Consulting is a bit like used car selling. If you're not playing close to the line, then someone else is.
Which, okay. Is what it is. But every customer should be aware this is what IBM et al.'s business model is, unless you're simply buying bodies.
Maybe I'm just naive, but I really think that strand still exists in IBM's corporate DNA. They still see this as solvable given sufficient time and effort.
When you realize how much positioning is required for large scale sales, you can start to understand why the same company might not have the feedback loops in place to build a product.
100% correct. Had IBM marketed Watson for what it actually is - an enterprise version of Amazon Alexa or Google Assistant (which admittedly didn't even exist at the time) - they'd be in a completely different spot. Instead we got the Bob Dylan commercials (which under the covers was just speech to text, a hardcoded dialog tree, and text to speech) and all the "Watson is curing cancer" nonsense (which under the covers was just a rules engine sitting on top of a search engine intended to provide customized treatment plans). The actual technology works, but it's an NLP platform for building chatbots and search engines (think intent classifier, relevancy ranker, named entity/semantic relationship/sentiment annotators) not the AGI that it was marketed as.
* Whatever the quality of the technology (which I personally never saw as that compelling) was wrapped up in terribly written research code, making it practically impossible to setup and use.
* The Jeopardy demo was made possible by the existence of a marked-up source of general knowledge (Wikipedia), a ready-made bank of questions and answers from past shows (j-archive.org), and the fact that practically anyone had the ability to curate more Q&A pairs. This is almost totally different than the medical use case where the knowledge is wrapped up in proprietary textbooks and papers and the only people able to curate training data are medical professionals.
https://www.quora.com/Why-isn%E2%80%99t-machine-learning-mor...
Skip to Jae Won Joe's answer to see a case study with a patient showing the interactive process. Then, especially look at ending questions about missing or deceptive data. Seems like their good diagnoses comes from a combination of domain data and expertly reading people in front of them. Machines suck at the part I emphasized. The data sets might not reflect a lot of stuff like that, too. Who knows.
It would take a paradigm shift before AI became useful. Going to watson.com and getting a prescription for antibiotics would be useful and technically feasible. So would augmenting lower trained people to be able to deliver more care. Neither are possible legally and it's not exactly a field you can get away with disregarding the law.
* Procure patient medical files. This rabbit hole includes things like HIPPA compliance and sufficiently anonymizing the files. Also, hospitals consider 3 years of information on 100 patients to be a lot of data.
* Have trained medical professionals annotate the files. Good luck convincing doctors to do this. Also good luck in determining what to do if the handful of doctors you're granted a few hours with a week can't agree on annotations.
* Actually produce a model that is reasonably good at pulling information out of patient charts,
Even after all the above, you still only have a solution that is giving you structured data, which is something that other EMRs can provide w/o the problems of accuracy & completeness of NLP process. There is still the process of diagnosis and treatment and also the fact that this is all a loop ('active learning').
I mean, GPT-2 still often produces nonsense more similar to dream imagery than useful reasoning, but I gather most medical residency students are half-asleep most of the time anyway, so... :)
Is there any example of gpt-2 in the wild that is not nonsense?
You could point it at your data and it would tell you the answer before you'd even thought of a question.
No wonder we're disappointed.
All hype and marketing built on open source tools, all while AWS, Google, and Azure build out bigger cloud offerings. As OP mentioned, they used to be a great company.
If your answer is mission critical, probabilistic ML isn't advanced, explainable, or reliable enough for your problem.
If your answer is of the great-to-be-right, meh-to-be-wrong sort (e.g. ranking movie recommendations), then you can and should go nuts with ML.
And if someone really wants to do an ML project on the former, do everything you can to transform it into the latter.
It seems like a problem when classifying user-provided images (eg. identifying obscene images on a social network) but not so relevant when you own the sensors.
But some projects as presented don't have a clear refusal option.
ML would go a long way if the Hippocratic Oath (or derivative thereof) were taken & adhered to by the industry.
After using that, I realized that Watson was mostly a gimmick.
https://twitter.com/chaoticmass/status/677708888083439616
I did look at the BlueMix platform, and it seemed like a big mix of various APIs I could tap into. Some of them looked neat, but I never had a reason to really use any of them.
The only explanation I've been able to come up with is that when it's all human processed, Joe 2nd-link-in-the-chain just deals with all the inconsistencies as best he can to get his job done, and never reports issues up.
It generally seems like (a) the suggestions for fixes are impractical to implement (overly detrimental effect on counterparty), (b) Joe isn't empowered organizationally to suggest fixes that will be implemented, or (c) Joe doesn't have access to the IT tools to implement fixes himself.
Finally got my hands on it and was incredibly disappointed. Lacking features, archaic interface, and not delivering on what it was promised. Talked to some guys working for the Watson team and they told me that half of the AI stuff is just them doing it manually under the guise of "integration and configuration".
I did have fun "talking" to Watson whenever I was bored at work, trying to make him swear