Companies will pawn off thier customers to AI at their peril.
In many ways this is a repeat of the India call center train wrecks of the 00s. On paper, letting someone in Bangalore vs onshore handle incoming customer service calls looked like a path to amazing savings. In practice the customer experience was horrendous and companies CTRL-Zed these decisions and rapidly brought customer service back onshore again. AI is just that story of shortsighted decisions by weak leadership playing out all over again.
I think of this as the kiosk economy. It's like how businesses introduce kiosks even when having a human employee would be more convenient, just for the sake of greater profit.
The biggest problem with AI customer service is that a human employee would've let a minor issue slide without escalating it. But a chatbot often inflames the situation, and by the time the customer reaches a human agent, they're already furious.
Most people aren't rational or logical. Non verbal feedback, like acknowledging someone's anger and showing empathy, is incredibly important.
if you've tried to legitimately productionize LLMs to deliver a specific repeatable outcome, rather than human editable output(text, code), you know the models still aren't there yet and the whole product experience feels hacky
As someone who's been doing AI call centres for a while, I'm not surprised. It's really, really hard to build voice AI that works. There are no open source solutions that work out of the box.
But the biggest problem is ASR. WER is still atrocious even with SOTA models. When you add drug names and regional accents, it's a recipe for disaster.
The desired speed of AI adoption is what is hindering AI adoption. Big companies are trying to sell this as magic, and it's not. It requires proper use to get anything useful out of this system. But that takes away the magic of it and the investors can't have that as the eventual share price only works if it is magic. Frontier labs will smother their product with their timelines.
It's important to note while people are pumping the AI bubble that a near-moron with a checklist can do this job. They replaced AI with a regular old phone tree.
AI couldn't do this job with a checklist. It would forget the entire checklist after two responses, then get fixated on the words that the customer used to complain about how dumb the AI was and loop until they hung up and called back to hopefully clear the context.
edit: Unless Kinney had been sold a product that used caller ID to carry over context, then the customer's best chances would be to scream "Agent!" over and over again (to get put on hold for 45 minutes, then transferred to the most screamed-at Philippine minimum-wage gig worker on the planet), or to show up to the drugstore and start screaming at people.
Pharmacy phone lines are a terrible first use case for voice AI. Older callers, drug names, insurance mess, zero patience. Automating "store hours" is fine. Automating refill exceptions is how you get this outcome.
So - I run a company that does AI agents for pharmacy. We are at Series B now and growing well - series C in 9-12 months given growth. We just acquired a competitor of that vendor so very much an insider.
First, The technology works, and it scales, but the whole bottleneck is domain expertise and implementation. These are expensive, and hard to scale. We hire pharmacists as project managers, that's how important domain expertise and implementations are.
Second, the amount of noise of "Voice AI for <industry>" is incredible. Most of them are completely clueless about the industry & basically "YC-striver" type who can only sell to other yc companies. They fail hard the moment they touch critical functions of the real world.
We have hundreds of pharmacies with us now and they mostly successful and happy. Our biggest friction is customers being burned by vendors like this and writing off AI altogether as "not ready".
FWIW I know of a pharmacy network that's implementing AI. Not only does the AI make egregious errors like speaking in Spanish when the conversation is in English. It also has a limited context window for ground rules of 5000 tokens or less. So correcting all the various errors goes over the instructions limit. As a result if the implementation goes through it will be of poor quality. All I can think is that people with no technical expertise are making these decisions, it is easy for any techie to predict that AI of this level won't be capable.
> wrong dosages, and missed prescription notifications.
BS. How is the AI touching that information?
Prescription notification is outbound communication. It’s almost a recorded message. A little text to speech if it is supposed to read the drug name.
Wrong dosages?? This must be handled by the “enroll a new scrip” functionality, right? So basically B2B doctor interaction. You let AI touch that? Aren’t the vast majority of the costs from customers not doctors? None of this makes sense.
I happened to be in Vermont last week and was unaware of this change. I heard complaints from three separate people unprompted. It was a total dumpster fire put on top of a critical service.
Do we think that as a society there exists some meaningful lever by which we can apply force against this type of carelessness and exploitation?
I think we need to take a good hard look at what we want out of this world and ask ourselves. Do we really need to automate this? Like really. What benefit does the world receive having an AI chatbot tell grandma the wrong doage for her meds? This is the naked extraction of value from a service that's already been offshored. Companies like these are gleeful to make their service as shitty as they can because they know their users are doing so against their will. Infact, that's become the goal.
my experience with AI assistants has not been that great so far, they ask for customer id numbers which are long as ... with several digits and if you group them they don't understand, for example if the number begins with two 1s and you say eleven instead of one one they don't understand you, then they are usually not trained for the local pronunciation or dialect which makes it very hard communicating, my blood pressure rises into unhealthy ranges, but if I use expletives enough I get a human on the line
wonder how much of the ai CX hate is availability bias where people really notice it when it breaks but you never see the whole picture on total impact
it seems very implementation dependant, I know a startup with incredibly realistic voice agents that most people don’t detect is an ai that outperforms humans. But ai for CX seems like an easy project to do for companies looking to cut costs so the average agent a user encounters in the wild is poor quality
Not Pharmacy, but I literally had Shopify checkout button clickjack me and then AI agent and human agents (chat, email, and phone) refuse to cancel the resulting order today. Despite contacting within 5 minutes. Given how hostile so many companies are, I question why they even bother appearing to have support.
But this doesn't seem likely to translate well to pharmacy, where there will be consequences for treating customers like vermin, as opposed to actual incentives to do it.
I wouldn't be shocked if the tech needed in places like pharmacy or other sectors where there's some level forced responsibility leads to the version that's actually palatable elsewhere.
Cases like this raise a question. As an engineer, the errors that coding AIs make are an annoyance, and reassuring in terms of my job prospect.
As a consumer, the errors that AIs make are more than an annoyance. I have yet to meet an AI phone assistant that can do anything more than an explicitly programmed phone tree, and several, including the one for the pharmacy I use, that do far less. They are undoubtedly much more expensive to create and test.
Clearly there's a bubble going on, and unprincipled people chasing funding and promotions as usual, but why are companies like CVS paying more to get less?
I think it is literally to waste our time. It's one more layer of defense to stop you from talking to a person.
There are two big wins in health care. One is to increase the scope of health care that gets paid for. More and more expensive treatments, to bring the money in. The second big win is to stop people from getting treatment. Deny coverage, confuse them and wear them out with bureaucracy, and in this case stop them from talking to an expensively educated human pharmacist. That's what an AI does better than a phone tree, because it more effectively creates a sense of helplessness and valuelessness in the customer.
Expanding the scope of care you're entitled to and then fighting against your ability to get it isn't a contradiction any more than it's a contradiction when a person breathes in and breathes out. It's two synergistic parts of a system whose goal ultimately has nothing to do with health care.
All of these conversational AIs should either confidently detect a users intent or apologize and offer to forward call to a real person.
LLMs perform pretty poorly when you start try to restrict them too much (e.g. detect intents and control responses based on intent detection), the replies gets disjointed and weird feeling. An open ended prompt and chat completion gives a much better feeling most of the time but can go off the rails.
If you don't let LLMs handle the conversation entirely you get into insanely complex traditional engineering problem: (if user is complaining do x, if user asks a relevant question do y, if user asks an irrelevant question do z), which is an endless can of worms.
The other win from obstructing people's access to healthcare is extending ill health and hence demand for the product. The trick is to not hasten end of the demand through death, but I'm sure that's within the ability of a simple "AI".
Getting the run a round from the Duke Medical School phone system was the last straw for my late wife. She killed herself because she could not reach a Human that could help her.
So yes, death does happen by phone systems.
This was in 2013 pre AI.
See our documentary Pain Warriors about that whole saga.
Free to watch on TubiTV and Amazon Prime.
I get no renumerations of any kind.
The opposite actually, has cost me greatly.
Pain Warriors has won several awards.
Actual personal service from a human being is the main reason that I'd choose a small regional pharmacy over a local CVS or mail-order. It seems to me like personal service is the ONLY advantage a regional chain like this has. If I were them, I'd lean into it rather than burning customer's goodwill with AI.
38 comments
[ 1.8 ms ] story [ 21.3 ms ] threadWe need to fight the talking point that open source is bad!
In many ways this is a repeat of the India call center train wrecks of the 00s. On paper, letting someone in Bangalore vs onshore handle incoming customer service calls looked like a path to amazing savings. In practice the customer experience was horrendous and companies CTRL-Zed these decisions and rapidly brought customer service back onshore again. AI is just that story of shortsighted decisions by weak leadership playing out all over again.
The biggest problem with AI customer service is that a human employee would've let a minor issue slide without escalating it. But a chatbot often inflames the situation, and by the time the customer reaches a human agent, they're already furious.
Most people aren't rational or logical. Non verbal feedback, like acknowledging someone's anger and showing empathy, is incredibly important.
But the biggest problem is ASR. WER is still atrocious even with SOTA models. When you add drug names and regional accents, it's a recipe for disaster.
AI couldn't do this job with a checklist. It would forget the entire checklist after two responses, then get fixated on the words that the customer used to complain about how dumb the AI was and loop until they hung up and called back to hopefully clear the context.
edit: Unless Kinney had been sold a product that used caller ID to carry over context, then the customer's best chances would be to scream "Agent!" over and over again (to get put on hold for 45 minutes, then transferred to the most screamed-at Philippine minimum-wage gig worker on the planet), or to show up to the drugstore and start screaming at people.
First, The technology works, and it scales, but the whole bottleneck is domain expertise and implementation. These are expensive, and hard to scale. We hire pharmacists as project managers, that's how important domain expertise and implementations are.
Second, the amount of noise of "Voice AI for <industry>" is incredible. Most of them are completely clueless about the industry & basically "YC-striver" type who can only sell to other yc companies. They fail hard the moment they touch critical functions of the real world.
We have hundreds of pharmacies with us now and they mostly successful and happy. Our biggest friction is customers being burned by vendors like this and writing off AI altogether as "not ready".
https://arstechnica.com/tech-policy/2024/02/air-canada-must-...
> Air Canada essentially argued that “the chatbot is a separate legal entity that is responsible for its own actions”
BS. How is the AI touching that information?
Prescription notification is outbound communication. It’s almost a recorded message. A little text to speech if it is supposed to read the drug name.
Wrong dosages?? This must be handled by the “enroll a new scrip” functionality, right? So basically B2B doctor interaction. You let AI touch that? Aren’t the vast majority of the costs from customers not doctors? None of this makes sense.
I think we need to take a good hard look at what we want out of this world and ask ourselves. Do we really need to automate this? Like really. What benefit does the world receive having an AI chatbot tell grandma the wrong doage for her meds? This is the naked extraction of value from a service that's already been offshored. Companies like these are gleeful to make their service as shitty as they can because they know their users are doing so against their will. Infact, that's become the goal.
Set up whatever online portal or app you want. I’d prefer to just sign in and click a button.
If I’m picking up the phone or sending an email it’s because I have a weird problem and I need to talk to a person.
it seems very implementation dependant, I know a startup with incredibly realistic voice agents that most people don’t detect is an ai that outperforms humans. But ai for CX seems like an easy project to do for companies looking to cut costs so the average agent a user encounters in the wild is poor quality
But this doesn't seem likely to translate well to pharmacy, where there will be consequences for treating customers like vermin, as opposed to actual incentives to do it.
I wouldn't be shocked if the tech needed in places like pharmacy or other sectors where there's some level forced responsibility leads to the version that's actually palatable elsewhere.
As a consumer, the errors that AIs make are more than an annoyance. I have yet to meet an AI phone assistant that can do anything more than an explicitly programmed phone tree, and several, including the one for the pharmacy I use, that do far less. They are undoubtedly much more expensive to create and test.
Clearly there's a bubble going on, and unprincipled people chasing funding and promotions as usual, but why are companies like CVS paying more to get less?
I think it is literally to waste our time. It's one more layer of defense to stop you from talking to a person.
There are two big wins in health care. One is to increase the scope of health care that gets paid for. More and more expensive treatments, to bring the money in. The second big win is to stop people from getting treatment. Deny coverage, confuse them and wear them out with bureaucracy, and in this case stop them from talking to an expensively educated human pharmacist. That's what an AI does better than a phone tree, because it more effectively creates a sense of helplessness and valuelessness in the customer.
Expanding the scope of care you're entitled to and then fighting against your ability to get it isn't a contradiction any more than it's a contradiction when a person breathes in and breathes out. It's two synergistic parts of a system whose goal ultimately has nothing to do with health care.
LLMs perform pretty poorly when you start try to restrict them too much (e.g. detect intents and control responses based on intent detection), the replies gets disjointed and weird feeling. An open ended prompt and chat completion gives a much better feeling most of the time but can go off the rails.
If you don't let LLMs handle the conversation entirely you get into insanely complex traditional engineering problem: (if user is complaining do x, if user asks a relevant question do y, if user asks an irrelevant question do z), which is an endless can of worms.
So yes, death does happen by phone systems.
This was in 2013 pre AI.
See our documentary Pain Warriors about that whole saga. Free to watch on TubiTV and Amazon Prime. I get no renumerations of any kind. The opposite actually, has cost me greatly. Pain Warriors has won several awards.