- remember when search first came out and websites often had their own search bar, how bad it was and how irrelevant the top answers often were. I think chat like this gets picked on more because of the natural language element, when it's not necessarily worse than previous tech
- I understand the negative press about these "failures" or flaws but it's unfortunate to see innovation discouraged. Building these things and putting them in the wild is the best way to test them, and this is still pretty low risk. They should be applauded for trying, it's their reaction that's more important. (See air canada fighting it's customers in court instead of refunding a few hundred buck over a chatbot mistake)
- related, a idea would be a kind of sandbox convention where it's made clear these are works in progress and will have issues. I don't know if that will satisfy everyone but at least it will make the readiness of the technology clear
- I hope orgs don't get discouraged from trying to build these because of the bad press
To your top point, providing a wrong answer is significantly worse than providing an irrelevant one because the latter can be easily identified as such by the requester.
Getting no results is a dramatically better outcome than the LLM behavior of making up something named “ibond” and providing information about this thing that doesn’t actually exist.
The parallel I think of is the dark UX employed by every streaming app. If it doesn't have the show I explicitly searched for, it shows me a "you might be interested in" list that takes up the majority of the screen, instead of a simple "No results matched your query".
> when it's not necessarily worse than previous tech
This is a good point, but on the other hand I think there is a notable difference between “the tool makes it hard to find the page with the correct info you need” and “the tool confidently gives you incorrect information without making it easy to audit accuracy”.
I see a big difference between one side searching for human written content and not finding it and on the other side just making up stuff. Maybe most people expect the AI to search and quote from existing human-authored content? At least in situations where laws and regulations are involved this should be the default.
The problem will only get worse and you can expect humans to copy those AI-authored answers into the web where AIs are likely to digest them as facts.
If there are any short story writers out there, you could create an entire genre from ideas about hallucinating AIs. Marathon did it as "rampancy" back in the 90s but this is arguably even more interesting.
If an AI chatbot cannot reliably produce correct information in matters of law/compliance like this, I do not think it should be permitted to be presented to regular users in any fashion, beta or not.
It is especially egregious when it may be presented on a website that is meant to serve as an authoritative and reliable source of truth being that it is run by a government entity.
We already know plenty of people will just ignore the warnings of not taking its results at face value and just trust what it says, since they will state things authoritatively that are simply not true.
Indeed. A commenter up there said this is a "low-risk" application. Tech folks don't seem to understand that there are less tech-literate people out there taking these answers at face value, and being worse off for not knowing their rights.
>Tech folks don't seem to understand that there are less tech-literate people out there taking these answers at face value, and being worse off for not knowing their rights.
Tech folks don't care, there's too much money to be made in the grift.
This is exactly it! I’m not even that smart and I know that these LLM can never get to the point of being 100% accurate, it’s inherent to how they work. So if I know that, certainly all of the experts working on these systems know that. So many mendacious f*cks.
More examples from the original article, which goes into more detail:
"The bot said it was fine to take workers’ tips (wrong, although they sometimes can count tips toward minimum wage requirements) and that there were no regulations on informing staff about scheduling changes (also wrong). It didn’t do better with more specific industries, suggesting it was OK to conceal funeral service prices, for example, which the Federal Trade Commission has outlawed."
The problem is that legal advice is hard and requires nuance. It's not even entirely wrong. Simply not paying rent is not sufficient cause for eviction. Not paying rent is permissible if maintenance is neglected.
Despite IBM's colorful moral history in the early days of computing, IBM had it right: "A computer can never be held accountable, so has increasingly been used to make management decisions". I'd extend that to "human" or "important" decisions.
20 comments
[ 3.5 ms ] story [ 61.1 ms ] thread- remember when search first came out and websites often had their own search bar, how bad it was and how irrelevant the top answers often were. I think chat like this gets picked on more because of the natural language element, when it's not necessarily worse than previous tech
- I understand the negative press about these "failures" or flaws but it's unfortunate to see innovation discouraged. Building these things and putting them in the wild is the best way to test them, and this is still pretty low risk. They should be applauded for trying, it's their reaction that's more important. (See air canada fighting it's customers in court instead of refunding a few hundred buck over a chatbot mistake)
- related, a idea would be a kind of sandbox convention where it's made clear these are works in progress and will have issues. I don't know if that will satisfy everyone but at least it will make the readiness of the technology clear
- I hope orgs don't get discouraged from trying to build these because of the bad press
If you don't know that, you get horribly confused.
This is a good point, but on the other hand I think there is a notable difference between “the tool makes it hard to find the page with the correct info you need” and “the tool confidently gives you incorrect information without making it easy to audit accuracy”.
The problem will only get worse and you can expect humans to copy those AI-authored answers into the web where AIs are likely to digest them as facts.
It is especially egregious when it may be presented on a website that is meant to serve as an authoritative and reliable source of truth being that it is run by a government entity.
We already know plenty of people will just ignore the warnings of not taking its results at face value and just trust what it says, since they will state things authoritatively that are simply not true.
But with these city bots, airline bots, Alexa, etc., getting it wrong could have dire real life consequences.
Tech folks don't care, there's too much money to be made in the grift.
More examples from the original article, which goes into more detail:
"The bot said it was fine to take workers’ tips (wrong, although they sometimes can count tips toward minimum wage requirements) and that there were no regulations on informing staff about scheduling changes (also wrong). It didn’t do better with more specific industries, suggesting it was OK to conceal funeral service prices, for example, which the Federal Trade Commission has outlawed."
Here's a direct link to the list of wrong answers from that article: https://themarkup.org/news/2024/03/29/nycs-ai-chatbot-tells-...
There have been plenty of stories showing that it takes more than just being a squatter to get someone out.
Despite IBM's colorful moral history in the early days of computing, IBM had it right: "A computer can never be held accountable, so has increasingly been used to make management decisions". I'd extend that to "human" or "important" decisions.