FTA - "The problem is that healthcare giants’ primary obligation no longer appears to reside with patients, but with their financial results", that, in a nutshell, is the core of everything wrong with US healthcare: it focuses first on shareholders, and secondarily on patients. It's no surprise the US spends more on healthcare per capita than any other country in the world and yet gets far worse outcomes.
“Everyone always acts in their own self-interest” is a hyperstition — if enough people believe it, this will become true. But it is also possible that most doctors go into medicine to cure patients and that most nurses want to do the job they signed up for. That humans are inherently social creatures who have intrinsic desires which can be things other than personal financial gain, and that the social context in which they experience these desires affects whether it’s a “good idea” to act on them. You are discarding every bit of pure altruism, every bit of pride in a job well done, every bit of contributing to your community for the good of all.
The rise of the American administrative sociopath is neither a fait accompli nor a laudable end-goal.
Can you show me a single situation in which this has happened at scale? Has any communist country not had major productivity problems? The explanation is simple, really. Social groups were never meant to be larger than a town, and human empathy breaks down when considering millions people.
Did I ever say the U.S system is perfect? Do you think other countries have no problems? You people focusing on "people over profits" is literally communism, and is worthless. You're never going to get people to care about others beyond a superficial show of solidarity. That's why I said we need to align people's outcome to profit. Do you think that means an ancap healthcare system?
And nice comic. I completely agree that your attempts to change human psychology will fail.
No one said other countries don't have problems with their healthcare, what has been said and is demonstrated with ample data is they're getting far better health outcomes for far less money.
The 2nd half of Adam Smith's book was a treatise in all the different ways markets would fail. Adam Smith identified so many failure modes that the Founders thought Capitalism would never work (regardless of what the history revisionists now say). Healthcare is one of those verticals for which markets demonstrably don't work.
> At scale people will always act in their own self interest
That assumes self-interest == money.
Doctors and nurses go into the profession because they're interested in the work. So their self-interest is a combination of "good work" + "money".
Like there's easier ways to make doctor-level pay than going to medical school and residency for most of a decade, and graduating with nearly a half-million in debt. FAANG, finance, big law. So there are more incentives here than mere dollar bills. You don't have to be a "commie" to see this.
Never said it was. People go into creative fields because they enjoy it. That's called supply and demand.
Putting aside doctors, do you know we have a massive nursing shortage? Becoming a nurse is relatively easy, jobs are plentiful, and it pays really well. But it turns out that wiping the ass of a psychotic homeless man isn't something people enjoy. Realistically the shortage would be even worse if the job didn't pay well.
It's interesting how "AI" has become synonymous with "LLM" in people's minds.
There are many classifier models that have real value add. Catching potential mistakes or omissions in analysis has been well received (by docs and admins).
Be a tool to help doctors do a better job, help them deal with their increasing workloads.
When 100% of liability for malpractice is placed upon the senior execs and major shareholders of the companies pushing medical AI my skepticism will abate slightly. Until that happens you've got perverse incentives and no downsides to provide meaningful guardrails.
Usually when something's value and limitations has been proven people stop referring to it as AI. AI is used as a marketing term to generate hype for new thing.
> “No computer, no AI can replace a human touch,” said Amy Grewal, a registered nurse.
I don't believe this to be true in the long run. We will almost certainly have Westworld like robots in the future. Researchers have also found, on average, ChatGPT has better bedside manners than doctors.
Step 3.5 Doctor rewrites analysis to go with whatever the AI spits out because:
* Too busy (assumptions were made that patient load could increase with ML support)
* Too scared of being sued even if the doctor disagrees
* Too scared of being fired because the doctor keeps overriding the "cost effective" ML verdicts
We've seen this play out over and over in the medical field. I've watched my Primary Care physician go from angst over patient outcomes to angst over statistical outcomes for his department. It took 15 years, but the system got him.
True and if they're leading with their fucking feels when making major life decisions their arrested emotional development is liable to land them in deeply regrettable circumstances.
... even if the tech materializes, is Westworld the thing you want to reference if you're arguing that AI replacing humans is a good thing? I think all incarnations of that book/film/show involve robots that (a) don't work as intended and (b) kill people (c) make humans behave badly.
A quick glance at crash statistics for the US suggests that no meaningful advances have been made in automobile safety in the last 20 years give or take. This assumes you accept fatalities per driver hour as a valid metric.
I feel like we're arguing the same side here? We didn't get mandatory in-console ADHD factories and 6000lb daily drivers that punch through DOT guardrails because they enhance safety.
This is a case where LLMs and similar generative approaches are outright dangerous and damaging. Not so much immediately but in that they erode trust in machine assisted diagnosis or treatment and set back adoption of things that could actually work substantially.
Namely tools like expert systems which are uniquely suited for the application can easily be confused with the current batch of "AI" despite being much more sane and useful.
I have a background in expert systems in a medical context, and it is wild to see people reaching for LLMs in this space. Having a clear audit trail for how a decision was reached is a requirement, and these tools don’t provide it.
LLMs can be a component in a system. See them as better language handlers than existing algos for looking at what doctors write to see if there are mistakes or omissions. Flag things for review rather than make decisions. In the audit trail, there is less concern about what model you used to flag something for human review. If you can catch mistakes, you can save lives, and the hospital money
This will be just like a self driving car system which I will not name, where you are handed a giant footgun with the disclaimer that you are responsible for any mistakes or hallucinations it might make.
Tech Bro: "Look, I told it I had a headache and it told me to take some Ibuprofen. Pretty soon you can fire all your doctors and nurses since DocGPT will do it all!"
And eventually: "Our other product, InsureGPT, has been specifically trained to identify legitimate expensive claims that can be denied to maximize the probability of exhausting the patient in the appeals process!"
This is so familiar it hurts. I worked on something which originally had the goal of giving doctors more time with patients, and it was slowly but surely steered towards being a useful tool for insurance companies. I was too naive to see it coming.
Whole areas of publishing are trying “have an expert write a book mostly with AI, then have an editor un-fuck it”. There are tons of businesses trying this “brilliant” plan right now.
From what I can tell, the labor savings is… low, and maybe zero. The output’s such trash, it takes enormous effort to clean it up. This stuff only seems to be a huge time-saver when someone doesn’t give any shits about how good the output is—so, mostly spam, scams, and astroturfing.
[edit] to temper the pessimism, it is really nice in small ways, in capable hands. But it’s not gonna be replacing many workers any time soon. Outside helping editors and writers with idea generation and little touch-ups, it’s great for non-writers who want to maintain a consistent tone in e.g. marketing copy, but requires the operator to have good taste and to be fairly thoughtful, which means they’ve got to be above-average at this stuff to begin with.
It's more a statement on the time than the technology, but at this moment GenAI was destined to become a force multiplier for laziness, grift, profit, and predation more so than forces that will make society better.
The second order effect is that because this effort is now possible every shadow or unknown contains this effort. Weirdly worded Reddit title? Must be an LLM. Poorly shot image? Must be StableDiffusion.
Well before GenAI, CStross has blogged about the value-add of editors being quite large.
I suspect he probably can't give a complete answer to the relative expense of human editors for human vs. AI writing given that he's said that the merest hint of GenAI in his profession would be a reputational death sentence.
LLMS, like cryptocurrency has now been discovered by the business bros with the dollar signs for eyes. This will make a big part of the world will get actively shittier and aside from actual cool and useful applications like summarization etc. I predict that the malicious usecases outweigh the good ones at least 10:1.
Anything which has clear paths to decision making, does not need AI.
Anything which requires only massive data processing, does not need AI.
Regarding the adoption: The crux is mentioned in the article well. You are building on top of faulty systems. However great the AI model is, you are given faulty data to process. It will not end well.
I would say the only reasonable and logical explanation is that these nurses are just salty that they have missed the AI stock craze and they just want to generate a dip for a buy-in. Gotta look out for those 4d chess moves. taps head
(/joke <- just in case)
"No computer, no AI can replace a human touch,” said Amy Grewal, a
registered nurse. “It cannot hold your loved one’s hand. You cannot
teach a computer how to have empathy.”
Sadly, this is terrible tactics. The people they are protesting do not see lack of empathy as a problem and probably even see it as a feature, as empathy is a bug from their point of view.
They'd be much better off talking about how obviously bad the care being given is and how easily people will be able to sue them for criminal negligence. And also, I've heard, you know, not that I'd ever do this, but I've heard some other nurses are even starting to tip off the patients that this is something they should sue over and giving them pointers on what to ask for during discovery. Certainly not something I'd ever do and I don't know any nurses personally who do this. I've just heard rumors. If you get my drift.
I'm not celebrating it, just calling it like it is.
Does anyone care what goes on in the mind of a person making a healthcare LLM product? I think we all know what they’re thinking: ride the hype train; get paid.
I think the nurse here is trying to appeal to human beings who are, more or less, afraid of dying alone.
One of Peter Lee's arguments in his AI in medicine book[1] is that the GenAIs (GPT4) actually excel at empathy. He gives a pretty compelling example where the GPT is able to empathize very well with a young girl who is having a medical issue. Empathy is part of the training set.
This strikes me as something that will fade over time, though. We will eventually learn to recognize fake empathy, just as once upon a time when a corporation said "Your business is important to us and we're trying to get a support person on the line for you as quickly as possible", it was believable and there was a good chance your customer believed it. Now of course we've all got a pretty good idea it's not true.
An AI can not empathize. We don't even really want it to; who wants to build an AI that "really" experiences losing a limb or losing a daughter? Not anyone I want actually building AIs. So this isn't even about whether they're "really conscious" or any of those somewhat tedious debates; even if they are human-level AI already they literally can't empathize. See the recent article where Meta's overly helpful AI yielded an answer as to how New York's public schools treated its disabled child. Even if the text was completely accurate it still had no standing to emit such text.
error-prone automation, programmed from the ground up to prioritize money over health, is incorrectly denying essential insurance coverage to the elderly.
I remember being interviewed once for an opportunity in a startup working in developing a product that using AI they would automate answering to incorrect insurance denials (which is a tedious process).
AI or not, the problem of incorrect denials is so bad that there is a market for it.
Yea there are cottage industries on top of cottage industries in the medical insurance space. First you have the first order: DR supplies service, bills patient’s insurance, then you have the hospitals who outsource all billing, those outsourced companies outsource the claims management, which outsource the claims response, and the exact same structure exists on the insurance company side. And I’ve simplified a lot… there are coder companies, insurance coder consultants, etc. it’s totally wild.
If you ever wonder why. US medicine is so expensive: it’s bec there are 100 hands all reaching for their piece of your $120 annual checkup.
Or put another way, HIPPA is nice and all, but by the time all is said and done, 8-10 pairs of strangers eyes have seen your vagina fungus infection pharmacy bill.
I once was lucky enough to have my primary care physician through a university teaching hospital that outsourced almost nothing. Billing was in-house, insurance ops and claims processing was in-house. Labs went to an accredited in-house reference laboratory.
I am sure it was easy to justify doing everything in-house because the university held accredited degree programs for virtually every healthcare area from medical school to ophthalmology and optometry, physical therapy, pathology, dentistry, a college of nursing, pharmacy school, as well as specialties like oncology, cardiology, and anesthesiology. For the areas that weren't medical disciplines, they operated degree programs for bioinformatics, hospitality (the hospital cafeteria was excellent), and several other areas that just so happened to address essentially all the needs of a modern healthcare organization.
It was a fantastic model, but it might have only made sense in the context of a large teaching hospital.
As a healthcare tech investor, I do see a lot of startups selling potentially dangerous AI systems into the healthcare system. That said, there are also a good number of companies that are implementing systems thoughtfully to address a number of issues that are very real in healthcare like staff burnout, continuing education, adherence to standard of care, managing complex value-based payment contracts and coordination of care, etc. The trouble I see is that clinician/hospital buyers of these systems can't always tell the difference. A basic initial filter that can be used is simply (a) does the team have an experienced medical professional with power on its executive team, and (b) does the team credibly know how to measure clinical quality impact of what they're building and do they have a plan to honestly measure it.
we are assuming that the state of health care technology somehow starts with high data integrity that isn't reckless to begin with anyways. As a technologist, it pains me to say that medical personnel having a healthy sense of interpretation and distrust for the system is a good thing.
I see a few of these AI note taking apps for healthcare and can’t help but shake my head.
This is just not how nurses work on the ward. Nurses are most worried about liability. So notes are written in a way to protect ourselves in addition to actual patient info. Probably more so now that I think about it.
I was working at a big tech company and was critical of using AI in healthcare because of the hallucination problem. I was quickly fired for being 'too negative' and 'not showing a growth mindset'.
If AI is telling the nurse the substance and quantity I should be injected with I am going to have some concerns. If it is filling forms and unburdening doctors and nurses from bureaucratic busy work then it is great.
Startups are treating LLMs like a silver-bullet that can be applied to every area. They are a specific tool that do very well at specific problems (for now).
Ultimately the substance and quantity you will be injected with also depends on those same bureaucratic busy work. If the LLM hallucinates a new condition for you when filling in some field, or if it gets your BP slightly wrong, you're in for a bad time.
Kaiser has been working with Stanford for years, to train different models, especially with images. A good use-case could be for diabetic retinopathy. There are more and more diabetics and this can affect vision over time, if not properly diagnosed.
Kaiser has ramped up and now takes more and more eye pictures to check for diabetic retinopathy. But it is a slow and manual process, requiring optometrists and ophthalmologists to seat and careful check pictures one by one (takes about 20min per picture). It is just possible to screen as many as we should screen. Here AI could really help, and pre-scan pictures for special attention, and be able to screen order of magnitude more patients.
I guess at the end it might be something like should we just screen 10 patients with a doctor, or pre-screen 10,000 in that same time, and have the doctor still see 10 patients ?
Where is the evidence that a half-cooked reckless AI or an LLM for that matter was rolled out in this hospital? It’s not like this is the first AI tech used in a medical environment. Patients deserve the best care possible regardless. Who would argue that the current system is working as best as it could?
85 comments
[ 3.5 ms ] story [ 136 ms ] threadThe rise of the American administrative sociopath is neither a fait accompli nor a laudable end-goal.
https://www.commonwealthfund.org/publications/issue-briefs/2...
If socialized healthcare makes a country communist, then I'd argue exchanging capital for goods makes China capitalist.
> The explanation is simple, really. Social groups...
https://xkcd.com/592/
And nice comic. I completely agree that your attempts to change human psychology will fail.
Not explicitly, but I'm reading between the following lines:
> That's why I said we need to align people's outcome to profit.
Where do you think the US rates from a "focused on profit" perspective?
The 2nd half of Adam Smith's book was a treatise in all the different ways markets would fail. Adam Smith identified so many failure modes that the Founders thought Capitalism would never work (regardless of what the history revisionists now say). Healthcare is one of those verticals for which markets demonstrably don't work.
That assumes self-interest == money.
Doctors and nurses go into the profession because they're interested in the work. So their self-interest is a combination of "good work" + "money".
Like there's easier ways to make doctor-level pay than going to medical school and residency for most of a decade, and graduating with nearly a half-million in debt. FAANG, finance, big law. So there are more incentives here than mere dollar bills. You don't have to be a "commie" to see this.
Putting aside doctors, do you know we have a massive nursing shortage? Becoming a nurse is relatively easy, jobs are plentiful, and it pays really well. But it turns out that wiping the ass of a psychotic homeless man isn't something people enjoy. Realistically the shortage would be even worse if the job didn't pay well.
AI doctor right now also doesn't suggest that all my medical issues are "are you sure you're not on your period?"
There are many classifier models that have real value add. Catching potential mistakes or omissions in analysis has been well received (by docs and admins).
Be a tool to help doctors do a better job, help them deal with their increasing workloads.
I don't believe this to be true in the long run. We will almost certainly have Westworld like robots in the future. Researchers have also found, on average, ChatGPT has better bedside manners than doctors.
1. Patient gets tests / imaging
2. Doctor does analysis
3. AI double checks
4. Doctor writes final report
5. AI communicates with more empathy
If you haven't had a dbag doctor, you might not know how an AI might be preferable to that experience
* Too busy (assumptions were made that patient load could increase with ML support)
* Too scared of being sued even if the doctor disagrees
* Too scared of being fired because the doctor keeps overriding the "cost effective" ML verdicts
We've seen this play out over and over in the medical field. I've watched my Primary Care physician go from angst over patient outcomes to angst over statistical outcomes for his department. It took 15 years, but the system got him.
People should have a choice
Sex bots are around the corner and driving the progress.
I'd be happy with a Star Wars droid for a doctor if it meant it cost less and I didn't spend more time waiting than being helped
Namely tools like expert systems which are uniquely suited for the application can easily be confused with the current batch of "AI" despite being much more sane and useful.
And eventually: "Our other product, InsureGPT, has been specifically trained to identify legitimate expensive claims that can be denied to maximize the probability of exhausting the patient in the appeals process!"
From what I can tell, the labor savings is… low, and maybe zero. The output’s such trash, it takes enormous effort to clean it up. This stuff only seems to be a huge time-saver when someone doesn’t give any shits about how good the output is—so, mostly spam, scams, and astroturfing.
[edit] to temper the pessimism, it is really nice in small ways, in capable hands. But it’s not gonna be replacing many workers any time soon. Outside helping editors and writers with idea generation and little touch-ups, it’s great for non-writers who want to maintain a consistent tone in e.g. marketing copy, but requires the operator to have good taste and to be fairly thoughtful, which means they’ve got to be above-average at this stuff to begin with.
New underground marketing campaign by Polydactyls Against Mutilation, or lazy excuse when you get caught using the machine?
I suspect he probably can't give a complete answer to the relative expense of human editors for human vs. AI writing given that he's said that the merest hint of GenAI in his profession would be a reputational death sentence.
But he posts here sometimes, so: *waves*
Anything which requires only massive data processing, does not need AI.
Regarding the adoption: The crux is mentioned in the article well. You are building on top of faulty systems. However great the AI model is, you are given faulty data to process. It will not end well.
They'd be much better off talking about how obviously bad the care being given is and how easily people will be able to sue them for criminal negligence. And also, I've heard, you know, not that I'd ever do this, but I've heard some other nurses are even starting to tip off the patients that this is something they should sue over and giving them pointers on what to ask for during discovery. Certainly not something I'd ever do and I don't know any nurses personally who do this. I've just heard rumors. If you get my drift.
I'm not celebrating it, just calling it like it is.
I think the nurse here is trying to appeal to human beings who are, more or less, afraid of dying alone.
[1] https://www.amazon.com/AI-Revolution-Medicine-GPT-4-Beyond
An AI can not empathize. We don't even really want it to; who wants to build an AI that "really" experiences losing a limb or losing a daughter? Not anyone I want actually building AIs. So this isn't even about whether they're "really conscious" or any of those somewhat tedious debates; even if they are human-level AI already they literally can't empathize. See the recent article where Meta's overly helpful AI yielded an answer as to how New York's public schools treated its disabled child. Even if the text was completely accurate it still had no standing to emit such text.
AI or not, the problem of incorrect denials is so bad that there is a market for it.
Who is the customer? Insurance companies?
If you ever wonder why. US medicine is so expensive: it’s bec there are 100 hands all reaching for their piece of your $120 annual checkup.
Or put another way, HIPPA is nice and all, but by the time all is said and done, 8-10 pairs of strangers eyes have seen your vagina fungus infection pharmacy bill.
I am sure it was easy to justify doing everything in-house because the university held accredited degree programs for virtually every healthcare area from medical school to ophthalmology and optometry, physical therapy, pathology, dentistry, a college of nursing, pharmacy school, as well as specialties like oncology, cardiology, and anesthesiology. For the areas that weren't medical disciplines, they operated degree programs for bioinformatics, hospitality (the hospital cafeteria was excellent), and several other areas that just so happened to address essentially all the needs of a modern healthcare organization.
It was a fantastic model, but it might have only made sense in the context of a large teaching hospital.
Nurses gather at Kaiser SF to protest AI in health care
https://www.nbcbayarea.com/news/health/nurses-kaiser-sf-prot...
A.I.’s impact on nursing and health care
https://www.nationalnursesunited.org/artificial-intelligence
Tbh, I had enough incorrect doctors in my life, I'd ALWAYS want to see what the AI says.
Many nurses already have an internal sense when a model might be “off” for a patient, so it’s easier to know when it’s predicting accurately or not.
(1) https://www.frontiersin.org/articles/10.3389/fmed.2023.12851...
I see a few of these AI note taking apps for healthcare and can’t help but shake my head.
This is just not how nurses work on the ward. Nurses are most worried about liability. So notes are written in a way to protect ourselves in addition to actual patient info. Probably more so now that I think about it.
Startups are treating LLMs like a silver-bullet that can be applied to every area. They are a specific tool that do very well at specific problems (for now).
Kaiser has ramped up and now takes more and more eye pictures to check for diabetic retinopathy. But it is a slow and manual process, requiring optometrists and ophthalmologists to seat and careful check pictures one by one (takes about 20min per picture). It is just possible to screen as many as we should screen. Here AI could really help, and pre-scan pictures for special attention, and be able to screen order of magnitude more patients.
I guess at the end it might be something like should we just screen 10 patients with a doctor, or pre-screen 10,000 in that same time, and have the doctor still see 10 patients ?