It says the problem is management understanding is failing, but its more they do not listen to their technical staff but instead read the equivalent of Cool Stuff Magazine, know their investors read it too, and that the next board meeting is going to revolve around asking them what their grand strategy for AI is.
It doesn't matter that they dont have one, frankly most of their data projects fail anyway and you just need one article published about your new vision to sell it for another six months your investor class.
Currently hugged so I can't read the article, but I can only wonder how this compares to the batting average of any given R&D effort. 20% of projects succeeding on a cutting edge technology might be pretty good, no?
And in a hype cycle many many more projects get off the ground that normally, outside of a hype cycle, wouldn't have ever received the requisite funding.
> First, industry stakeholders often misunderstand — or miscommunicate — what problem needs to be solved using AI.
So the #1 problem every startup faces.
> Second, many AI projects fail because the organization lacks the necessary data to adequately train an effective AI model.
This is interesting, and reinforces the trend towards hoarding data assets.
> Third, in some cases, AI projects fail because the organization focuses more on using the latest and greatest technology than on solving real problems for their intended users.
Makes sense, tough to pick an architecture or model to stick with when better options release weekly.
> Fourth, organizations might not have adequate infrastructure to manage their data and deploy completed AI models, which increases the likelihood of project failure.
Sounds like lack of capital.
> Finally, in some cases, AI projects fail because the technology is applied to problems that are too difficult for AI to solve.
So only a minority of cases? All in all this report seems to be saying "AI is promising but startups are still hard".
>> Second, many AI projects fail because the organization lacks the necessary data to adequately train an effective AI model.
> This is interesting, and reinforces the trend towards hoarding data assets.
Companies completely misunderstand where the data is suppose to come from and attempt to hoard user data. The issue with LLMs is that the problems they are currently best suited for require data generated by the company. Manuals, decision trees, guides, tutorials, expert knowledge in general and companies aren't producing that material, because it's expensive. Also if that data existed, then maybe they wouldn't need an LLM.
Tons of LLM implementations are poor attempts to cover up issues with internal processes, lack of tooling and lack of documentation (without which the LLM can't function).
> > Fourth, organizations might not have adequate infrastructure to manage their data and deploy completed AI models, which increases the likelihood of project failure.
> Sounds like lack of capital.
I actually think this is also an engineering problem, or at least a 'human capital' issue. The skillset for developing an AI model and the skillset for deploying a massive data-based product are highly different, but people who are good at the former often get press-ganged into doing the latter. This is kind of a capital problem (more money means maybe they can hire a second person to manage the operations), but I think it's also just a general lack of awareness that MLOps is really it's own thing. Especially when you're moving fast, tech-debt with these systems builds up really quickly (shockingly quickly). More money lets you hide these problems better, but IMO the solution is only going to come with time as people develop better and better best-practices for this type of project.
edit: There's a section in the full report called 'Too Few Data Engineers' that does a better job making this point. Everybody wants to make fancy AI models, nobody wants to be responsible for the 10K lines of uncommented Python and SQL you're using to build your test/train sets
>Everybody wants to make fancy AI models, nobody wants to be responsible for the 10K lines of uncommented Python and SQL you're using to build your test/train sets
I'm unfortunately the guy who that gets dumped on and it's the most hated part of my job. I've tried talking to the people who authored such atrocities but they refuse to acknowledge that's bad code and have huge egos about it and see any slam dunk tools like using a linter to be an impediment to their work.
Actually an additional point, If you read the report, the 80% failure rate claim does not come directly from this report, but links to the following article as it's source:
https://fortune.com/2022/07/26/a-i-success-business-sense-ai...
(https://web.archive.org/web/20240105144440/https://fortune.c...)
Which actually further quotes ostensible sources without linking:
"That’s borne out in a slew of recent surveys, where business leaders have put the failure rate of A.I. projects at between 83% and 92%."
Unless someone is able to track down the original source of this information I would treat it with great scepticism.
Isn't that better than normal? It used to be 90% of startups going belly up within 3 years, and out of remaining 10%, 9% becoming zombies and 1% having a proper exit? This looks more like Pareto 80/20 which is way better.
A few percentage points? Should mean nothing for vcs as they look to 10x profit. This is the time to go heavy with investing. Turn the stones others are afraid to turn because of spooky single digit percent interest rates. More likely to find your 10x now than when money was cheaper.
It's about in-house development on large companies...
So, that would make it about 2x worse than normal. What IMO, sounds way too good to be true.
(But then, I've seen AI projects being determined complete successes by having the same kind of result that would be considered failures on a normal product: being complete, but nobody using them.)
From the report 80% is "twice the rate of failure for information technology projects that do not involve AI" (https://www.rand.org/pubs/research_reports/RRA2680-1.html) so seems that a 20% hit-rate isn't actually that great. Possibly it's a quirk of how they're normalizing the success/failure count?
Most of those non-AI projects are likely using well-established practices and technologies. From that perspective a 20% success rate seems pretty good to me.
Sure that makes sense, overall "success of a technology product" seems like a very fuzzy thing to try and measure so I imagine one could spin the numbers basically however you want
I think that is the question, how much legitimate R&D is really going on here vs trying to shove an LLM into some random hardware and ship it?
Or shove an LLM in some app trying to solve a problem that it isn't capable of.
No doubt that creating these models are hard, having the data is hard. But how much of the AI startups is actually that vs just shoving the OpenAI API in something.
I think we’d need to dig into the 80% that failed and what kind of “AI project” they were. Is this really R&D? Or were you trying to insert AI into something that didn’t need it, and failed because no one is using your expensive and annoying “chat with us” popup that your VP insisted would keep your company competitive?
It's not a hit rate I think, it's a not yet failed rate. A lot of these were started just recently. And a lot is deep in red but is supported by the VC IV line. And by supported I mean totally. If a Word competitor needs to cut costs it theoretically can scale down and survive. If a neural network startup needs to cut costs it has to shut down operations, due to incredible price to train models and run inference.
Timing will play a role in how that number shakes out. AI projects often stacked huge investments and may not yet have had enough time to burn through all the cash or have it clawed back by investors.
There's so much LLM shovelware getting spammed here daily that I have a hard time believing 20% of all projects are succeeding. Are we even far enough into the LLM era though for bad projects to have run out of their borrowed time?
Is anyone else finding their company is asking teams to “insert ai everywhere any way you can”?
That’s a sign of a problem imho. The hype is so high the directives are to use ai everywhere regardless of fit. I’m a believer of ai but shoehorning it into everything as that currently boosts stock prices seems insane.
* blocking every known LLM url due to fear of leaking information to it
* not wanting to hire expensive data scientists for any in house development
I even asked an Engineering Manager at Meta how much their own team use Llama day to day to multiply their productivity. Their answer was they don't use it at all, and they weren't aware of any internal tooling to utilize it for work
This kind of fits the narrative of some of the Mag7 earnings calls where they more or less say "we aren't sure where the revenue will ever come from.. but its a game theory style arms race where we can't afford to NOT be there if someone figures out how to make revenue in the space".
So the big guys are buying GPUs, building out datacenter, developing & training models, etc.. just in case.
Maybe LLMs will change some niche dramatically, maybe it will reshape society, or maybe nothing.
More prior revolutionary developments end up like crypto, voice assistants, IoT, smart homes than the number that end up like smartphones, web, or the PC.
I think the only case I can think of where AI will revolutionize positively is self driving cars. Revolutionizing transport will have huge implication. The next thing would be robots, but that's just making people lazy in the household and replacing jobs. Crypto, voice assistants, IoT, smart homes, are bad examples are these have a great chance to grow more still. They will probably replace smartphones as smartphones did with PCs.
I semi-purposely left out self driving cars, and the revolution it has/is/will provide again.. remains to be seen. Waymo is nearly a 20 year old project at this point and is seemingly quite great in 2 cities, serving ~1% of the US population. These cities also happen to be in warm climates so theres a whole slew of environments / "edge cases" they just don't have to deal with. Maybe 20 more years?
So it's both outperformed what pessimists might have said (never work) and vastly underperformed what the median enthusiast projected (it's always just a few years away). I'd wager we are still teaching teens how to drive even 20 years from now.
LLMs are ~7 years old, so maybe another decade to being useful if we go by self driving cars learning rate?
Meanwhile at Google, 50% of code characters are from LLM autocomplete: https://research.google/blog/ai-in-software-engineering-at-g... Which is a little disconcerting. Maybe need to up my code review game. Also I don't personally use them at all - am I really missing out? Sometimes I wonder.
Yes. No one learned any lessons the last time around with "put everything on the blockchain".
Or maybe they did learn you can make a profit off of hype alone, but it's not making the end user or anyone else's life better as a result. Who cares - line goes up, people get promotions.
This is happening at my company. Thankfully I'm senior enough to push back on most of the requests to add AI as I still haven't found a good use case for it in our product.
that's expected behavior on any new wave right? I've seen the same with microservices, ORMs, SPAs, etc - "use this hammer any way you can!" - and with product trends (crypto, mobile apps, SoLoMo, etc). It's normal. Companies live and die on how well they surf hype cycles.
I recently had the opposite, where the CEO of the startup was killing almost all ideas of adding AI to their products. Perhaps AI is just polarizing. Some companies are jumping at the opportunity. But older startups like the one I was working for are being ultra-conservative with spend and are maintaining a wait-and-see attitude.
Common issue when new tech comes out. The people who know the tech, but not their companies business focus on the tech. Many of them will get promotions, and make their way up in the company. The company will lose likely millions, and the guy will leave to damage another company. If the company is lucky, another person who takes the time to understand the companies customers will come in... throw away the stuff their predecessor built, and solve some real problems. If the company is unlucky, they will double down on the over complicated solutions, and lose to a startup that ignored the sexy stuff and focused on the customer problem.
I work at a financial services company that is quite behind their peers tech-wise and watching the internal politics of AI has been fascinating.
Management seems to see this as their opportunity to catch up on the cheap.
Rather than having modern tech systems and properly staffed engineering department, let's just uh.. have non-technical people do AI hackathons! Also instead of automating excel jockey jobs with server side data pipelines, what if we.. you guessed it.. gave the excel jockeys AI!
I can imagine this playing out in a lot of industries where the underdogs think its a shortcut and yet..
Yes, I was on an internal project recently that wanted to use LLMs in a way that was appropriate to evaluate if changes between two versions of a text were semantically meaningful, and limited to that scope, it would've been a really valuable tool.
We had a directive from management to, for political reasons, use AI in the tool as much as possible to show how innovative and forward-thinking the company is. This led to a bunch of poorly-thought-out choices and while the project is in production and has internal users... I don't think it was particularly successful.
Not all of that is due to the "use AI" directive; there were also poor technology and deployment stack choices that made things overly complicated and cost us a bunch of time.
"There's a new technology out there that makes new things possible, let's explore whether it makes sense to integrate it into what we're doing?" - not only is that the correct attitude, in the long run it's the only attitude that keeps companies alive assuming they have exposure to tech.
See the internet/web revolution, the mobile revolution, etc.
This is answered. They only looked at projects that actually implement machine learning etc, and they did not look at projects that use ready to use models (the so called prompt engineering projects)
The problem is not if the 80% of them fail, but if of the remaining 20% you get several black swans that make profit skyrocket for the whole investment set.
The problem is if none of them, even the surviving ones, don't worth too much anyway. In that case those billions would had been wasted. But if you invested everything to just one player, and that player failed, then your whole bet failed.
It all depends if the rate of black swan is too low to bother with. There is probably a point where spending vc money on lottery tickets starts looking like the more pragmatic investment.
Pretty sure this is about AI projects, i.e. potentially career-ending failures, not failed AI companies which in a VC context have a high expected rate of failure.
That's bad management if it's seen as career ending failure. Proper management just does the ROI on the successful ones versus overall cost. And doesn't risk everything on them. Large tech companies test hundreds of model changes and rollouts per year in AB tests for that reason.
Anecdotally, I've never seen a project be a career ender. Most individuals have many projects under their belt, it's rare that an individual has bet the farm on an effort in such a way that others would not employ them.
When a project fails, the lessons are often valuable for the next project - when a project succeeds it can often just be do to market position.
I guess that perspective depends if you are a VC or a company trying to apply AI.
The VC expects to lose most of the time and hit it out of the park once in a while, for a large net gain.
For companies trying to automate or increase productivity with AI, there are unlikely to be any massively profitable winners that will make up for the failures, so too many failures is going to hurt.
I'm not sure the second statement is true - there are plenty of sectors I can think of which could be massive winners that would make up for the failures.
Obvious examples are things like AI-assisted language translation for books if you are a publisher, implementation of cancer screening technologies if you are a healthcare provider, AI-assisted drug discovery if you are a pharma firm, AI-assisted discovery if you are a legal provider, AI trading algo's if you are a hedge fund...
All of these could result in some pretty profitable winners. It's also the sort of thing that can result in 'losers' if you call it wrong - e.g. if there is a 20% chance of success and I don't invest and my competitors do, what happens if I am wrong? You don't want to be the Kodak of your industry. As an example - if I am the worlds biggest provider of call centres, do I really want to bet big that my industry will never be automated, or do I want to start investing now to prepare for that as a possibility?
The 20% chance of success isn't typically 20% chance of a moonshot paying off, but rather of a project being successful and achieving it's cost saving/productivity and user acceptance goals.
I don't think your Kodak example works very well here - they were a classic business school case of not realizing what business they were really in, but most uses of AI (whether one means LLMs, or something else like most of your examples) are going to be automation/productivity enhancement, not "pivotal" changes.
I would guess that the failure rate is worse for AI startups in general because of lower capital and experience. These failing experiments can only shell out billions for Nvidia’s shovels for so long before they have to start selling their shirts for lunch.
Sufficiently large companies should have a healthy “internal portfolio” of projects as well. Some safe bets/easy wins, some “this probably won’t work but if it does it’s game changing” projects as well. If they want to continue to exist in a competitive market they need to strike gold occasionally (which means hitting rock a lot of the time as well).
This movie is familiar…most of this summary in the Rand report applies/applied to any overhyped new technology. E.g., try substituting “NoSQL” for “AI” and see how well most of it reads.
154 comments
[ 1.8 ms ] story [ 220 ms ] threadIt doesn't matter that they dont have one, frankly most of their data projects fail anyway and you just need one article published about your new vision to sell it for another six months your investor class.
As corporate consulting America has a tendency to call any project, no matter how speculative, wasted if it didn’t succeed.
site appears to be down
Does not appear to be in archive.is
> Error establishing a database connection
A bit of irony that salesforcedevops Wordpress can’t manage the traffic from HN
> By some estimates, more than 80 percent of AI projects fail — twice the rate of failure for information technology projects that do not involve AI.
Also, given how early in the hype cycle we are, there are a lot of projects that haven't failed yet but will likely fail in the end.
Wow gosh. Where does that money go? It just evaporates?
> First, industry stakeholders often misunderstand — or miscommunicate — what problem needs to be solved using AI.
So the #1 problem every startup faces.
> Second, many AI projects fail because the organization lacks the necessary data to adequately train an effective AI model.
This is interesting, and reinforces the trend towards hoarding data assets.
> Third, in some cases, AI projects fail because the organization focuses more on using the latest and greatest technology than on solving real problems for their intended users.
Makes sense, tough to pick an architecture or model to stick with when better options release weekly.
> Fourth, organizations might not have adequate infrastructure to manage their data and deploy completed AI models, which increases the likelihood of project failure.
Sounds like lack of capital.
> Finally, in some cases, AI projects fail because the technology is applied to problems that are too difficult for AI to solve.
So only a minority of cases? All in all this report seems to be saying "AI is promising but startups are still hard".
> This is interesting, and reinforces the trend towards hoarding data assets.
Companies completely misunderstand where the data is suppose to come from and attempt to hoard user data. The issue with LLMs is that the problems they are currently best suited for require data generated by the company. Manuals, decision trees, guides, tutorials, expert knowledge in general and companies aren't producing that material, because it's expensive. Also if that data existed, then maybe they wouldn't need an LLM.
Tons of LLM implementations are poor attempts to cover up issues with internal processes, lack of tooling and lack of documentation (without which the LLM can't function).
I'd say 80% has failed, so far.
> Sounds like lack of capital.
I actually think this is also an engineering problem, or at least a 'human capital' issue. The skillset for developing an AI model and the skillset for deploying a massive data-based product are highly different, but people who are good at the former often get press-ganged into doing the latter. This is kind of a capital problem (more money means maybe they can hire a second person to manage the operations), but I think it's also just a general lack of awareness that MLOps is really it's own thing. Especially when you're moving fast, tech-debt with these systems builds up really quickly (shockingly quickly). More money lets you hide these problems better, but IMO the solution is only going to come with time as people develop better and better best-practices for this type of project.
edit: There's a section in the full report called 'Too Few Data Engineers' that does a better job making this point. Everybody wants to make fancy AI models, nobody wants to be responsible for the 10K lines of uncommented Python and SQL you're using to build your test/train sets
I'm unfortunately the guy who that gets dumped on and it's the most hated part of my job. I've tried talking to the people who authored such atrocities but they refuse to acknowledge that's bad code and have huge egos about it and see any slam dunk tools like using a linter to be an impediment to their work.
Unless someone is able to track down the original source of this information I would treat it with great scepticism.
So, that would make it about 2x worse than normal. What IMO, sounds way too good to be true.
(But then, I've seen AI projects being determined complete successes by having the same kind of result that would be considered failures on a normal product: being complete, but nobody using them.)
https://web.archive.org/web/20240826091915/salesforcedevops....
Here's a link to the Rand report: https://www.rand.org/pubs/research_reports/RRA2680-1.html
I think that is the question, how much legitimate R&D is really going on here vs trying to shove an LLM into some random hardware and ship it?
Or shove an LLM in some app trying to solve a problem that it isn't capable of.
No doubt that creating these models are hard, having the data is hard. But how much of the AI startups is actually that vs just shoving the OpenAI API in something.
> By some estimates, more than 80 percent of AI projects fail — twice the rate of failure for information technology projects that do not involve AI.
So 60% general success rate vs 20% for an emerging technology that doesn't really have established best practices yet? That seems pretty good to me.
ai is still incredible tho.
That’s a sign of a problem imho. The hype is so high the directives are to use ai everywhere regardless of fit. I’m a believer of ai but shoehorning it into everything as that currently boosts stock prices seems insane.
* blocking every known LLM url due to fear of leaking information to it
* not wanting to hire expensive data scientists for any in house development
I even asked an Engineering Manager at Meta how much their own team use Llama day to day to multiply their productivity. Their answer was they don't use it at all, and they weren't aware of any internal tooling to utilize it for work
So the big guys are buying GPUs, building out datacenter, developing & training models, etc.. just in case.
Maybe LLMs will change some niche dramatically, maybe it will reshape society, or maybe nothing.
More prior revolutionary developments end up like crypto, voice assistants, IoT, smart homes than the number that end up like smartphones, web, or the PC.
So it's both outperformed what pessimists might have said (never work) and vastly underperformed what the median enthusiast projected (it's always just a few years away). I'd wager we are still teaching teens how to drive even 20 years from now.
LLMs are ~7 years old, so maybe another decade to being useful if we go by self driving cars learning rate?
Quick skim of Meta engineering's blog only turned up this incident response tool they're "enhancing with AI": https://engineering.fb.com/2024/06/24/data-infrastructure/le...
Management seems to see this as their opportunity to catch up on the cheap.
Rather than having modern tech systems and properly staffed engineering department, let's just uh.. have non-technical people do AI hackathons! Also instead of automating excel jockey jobs with server side data pipelines, what if we.. you guessed it.. gave the excel jockeys AI!
I can imagine this playing out in a lot of industries where the underdogs think its a shortcut and yet..
We had a directive from management to, for political reasons, use AI in the tool as much as possible to show how innovative and forward-thinking the company is. This led to a bunch of poorly-thought-out choices and while the project is in production and has internal users... I don't think it was particularly successful.
Not all of that is due to the "use AI" directive; there were also poor technology and deployment stack choices that made things overly complicated and cost us a bunch of time.
They simultaneously fretted about compliance, and insisted we use an internal wrapper around AI tools, which was not ready for a long time.
That general pattern of security/compliance being at odds with every other part of management happens outside of AI of course.
Shoehorning? Definitely bad.
"There's a new technology out there that makes new things possible, let's explore whether it makes sense to integrate it into what we're doing?" - not only is that the correct attitude, in the long run it's the only attitude that keeps companies alive assuming they have exposure to tech.
See the internet/web revolution, the mobile revolution, etc.
A startup?
Integrating chat bot into your support page?
What is AI?
Titles like this are often click bait - but since the site is down can't tell.
The problem is if none of them, even the surviving ones, don't worth too much anyway. In that case those billions would had been wasted. But if you invested everything to just one player, and that player failed, then your whole bet failed.
When a project fails, the lessons are often valuable for the next project - when a project succeeds it can often just be do to market position.
The VC expects to lose most of the time and hit it out of the park once in a while, for a large net gain.
For companies trying to automate or increase productivity with AI, there are unlikely to be any massively profitable winners that will make up for the failures, so too many failures is going to hurt.
Obvious examples are things like AI-assisted language translation for books if you are a publisher, implementation of cancer screening technologies if you are a healthcare provider, AI-assisted drug discovery if you are a pharma firm, AI-assisted discovery if you are a legal provider, AI trading algo's if you are a hedge fund...
All of these could result in some pretty profitable winners. It's also the sort of thing that can result in 'losers' if you call it wrong - e.g. if there is a 20% chance of success and I don't invest and my competitors do, what happens if I am wrong? You don't want to be the Kodak of your industry. As an example - if I am the worlds biggest provider of call centres, do I really want to bet big that my industry will never be automated, or do I want to start investing now to prepare for that as a possibility?
I don't think your Kodak example works very well here - they were a classic business school case of not realizing what business they were really in, but most uses of AI (whether one means LLMs, or something else like most of your examples) are going to be automation/productivity enhancement, not "pivotal" changes.
See the actual report:
https://www.rand.org/pubs/research_reports/RRA2680-1.html