> Despite the rush to integrate powerful new models, about 5% of AI pilot programs achieve rapid revenue acceleration; the vast majority stall, delivering little to no measurable impact on P&L.
This summer, I built two very sophisticated pieces of software. A financial ledger to power accrual accounting operations and a code generation framework that scaffolds a database from a defined data model to the frontend components and everything in between.
I used ChatGPT substantially. I'm not sure how long it would have taken without generative AI, but in reality, I would have just given up out of frustration or exhaustion. From the outside, it would appear to any domain expert that at least three other people worked on these giving the pace at which they got completed.
The completion of those two were seminal moments for me. I can't imagine how anyone, in any field of information systems, is not multiples more effective than they were five years ago. That directly affects a P&L and I can't think of anything in my career that is even remotely close to having that magnitude.
I don't know what encapsulates an AI pilot in these orgs, and I'm sure they are massively more complex than anything I've done. But to hear 95% of these efforts don't have a demonstrable effect is just wild.
I'm arriving at the conclusion that deployments of LLMs is most suitable in areas where the cost of false positives and, crucially, false negatives are low.
If you cannot tolerate false negatives I don't see how you get around the inaccuracy of LLMs. As long as you can spot false positives and their rate is sufficiently low they are merely an annoyance.
I think this is a good consideration before starting a project leveraging LLMs
> The data also reveals a misalignment in resource allocation. More than half of generative AI budgets are devoted to sales and marketing tools, yet MIT found the biggest ROI in back-office automation—eliminating business process outsourcing, cutting external agency costs, and streamlining operations.
Makes sense. The people in charge of setting AI initiatives and policies are office people and managers who could be easily replaced by AI, but the people in charge not going to let themselves be replaced. Salesmen and engineers are the hardest to replace, yet they aren't in charge so they get replaced the fastest.
Nobody actually wants half the useless tools companies are coming up with because most of the solutions are not really novel. They are just wrapping an LLM.
It's kinda like what I realized with the meta Ray-Bans: I can have these things on my face, they can tell me the answer to virtually any question in 10 seconds or less.
But I, as a human, rarely have questions to ask. When you walk in to your local grocery store - you generally know what you want and where to find it.
A ton of companies are just gluing LLM text boxes into apps and then scratching their heads when people don't use them.
Why?
Because the customer wasn't the user - it was their boss and shareholders. It was all done to make someone else think 'woah, they are following the trend!'.
The core issue with generative AI is that it all works best when focused in a narrow sense. There is like one or two really clever uses I've seen - disappointingly, one of them was Jira. The internal jargon dictionary tool was legitimately impressive. Will it make any more money? Probably not.
> Because the customer wasn't the user - it was their boss and shareholders.
I'm starting to get asked, "Could AI help you do such-and-such faster?" At first I tried to explain why the answer is no, because such-and-such doesn't lend itself to what AI is good at. But I'm starting to realize I'm going to have to tell them I am using it and maybe give them an example once in a while, because they're hearing too much about its wonderfulness to believe there's something it can't help with. They're going to think I'm just being stubborn even though I tell them I'm not opposed to using AI where it makes sense. If that means the job actually takes a little longer to add in the part where I use AI to speed it up, they'll be happier.
> But I, as a human, rarely have questions to ask.
I realistically have between 10-100 questions I ask per day about things not immediately related to work. Double that if you include work based questions.
5% are succeeding. People are trying AI for just about everything right now. 5% is pretty damn good, when AI clearly has a lot of room to get better.
The good models are quite expensive and slow. The fast & cheap models aren't that great - unless very specifically fine-tuned.
Will it get better enough so that that growth rate in success pilots grows from 5% - 25% in 5 years or 20? Who knows, but it almost certainly will grow.
It's hard to tell how much better the top foundation models will get over the next 5-10 years, but one thing that's certain is that the cost will go down substantially for the same quality over that time frame.
Not to mention all the new use cases people will keep trying over that timeline.
If in 10-years time, AI is succeeding in 2x as many use cases - that might not justify current valuations, but it will be a much better future - and necessary if we're planning on having ~25% of the population being retired / not working by then.
Without AI replacing a lot of jobs, we're gonna have a tough time retiring all the people we promised retirements to.
> "“Every single Monday was called 'AI Monday.' You couldn’t have customer calls, you couldn’t work on budgets, you had to only work on AI projects.”"
> "Vaughan saw that his team was not fully on board. His ultimate response? He replaced nearly 80% of the staff within a year"
Being that this is Fortune magazine, it makes sense that they're portraying it this way, but reading between the lines there a little bit, it seems like the staff knew what would happen and wasn't keen on replacing themselves.
Calling fancy auto-complete a thing that can solve math and coding puzzles and translate between English and other languages, all better than most humans is just bonkers. Especially since those skills were outside of the grasp of "universal" computing machines for half a century.
You could even say that comparing glorified ifs, and fors calculator to capabilities of even today's AIs is laughable.
This is proof LLMs are viable and productive in my opinion. The baseline rate for business failure over 5 years is around 90%, so they say. With how much hype surrounds LLM wrapper startups this is still an astounding amount of novel business model creation.
At this rate, how is it better than pure random chance?
The article mentions 19-20 year old founders, focused on solving single user problems, were the successes.
The sample size is 300 public AI deployments and an undisclosed number of private in-house AI projects. And the survey seems to only consider business applications, as compared with end-user applications like media and software. That's significant but not definitive.
Isn't it more likely that existing problems with low hanging fruit, perhaps unpopular answers, that could be solved by leaning on "AI". And perhaps "AI" wasn't the key to success?
35 comments
[ 6.7 ms ] story [ 53.4 ms ] threadFor example, I heard that SAP has an 80-90% deployment failure rate back in the day, but don't have a citable source for it.
This summer, I built two very sophisticated pieces of software. A financial ledger to power accrual accounting operations and a code generation framework that scaffolds a database from a defined data model to the frontend components and everything in between.
I used ChatGPT substantially. I'm not sure how long it would have taken without generative AI, but in reality, I would have just given up out of frustration or exhaustion. From the outside, it would appear to any domain expert that at least three other people worked on these giving the pace at which they got completed.
The completion of those two were seminal moments for me. I can't imagine how anyone, in any field of information systems, is not multiples more effective than they were five years ago. That directly affects a P&L and I can't think of anything in my career that is even remotely close to having that magnitude.
I don't know what encapsulates an AI pilot in these orgs, and I'm sure they are massively more complex than anything I've done. But to hear 95% of these efforts don't have a demonstrable effect is just wild.
(How should I invest if I have this thesis)
If you cannot tolerate false negatives I don't see how you get around the inaccuracy of LLMs. As long as you can spot false positives and their rate is sufficiently low they are merely an annoyance.
I think this is a good consideration before starting a project leveraging LLMs
Makes sense. The people in charge of setting AI initiatives and policies are office people and managers who could be easily replaced by AI, but the people in charge not going to let themselves be replaced. Salesmen and engineers are the hardest to replace, yet they aren't in charge so they get replaced the fastest.
It's kinda like what I realized with the meta Ray-Bans: I can have these things on my face, they can tell me the answer to virtually any question in 10 seconds or less.
But I, as a human, rarely have questions to ask. When you walk in to your local grocery store - you generally know what you want and where to find it. A ton of companies are just gluing LLM text boxes into apps and then scratching their heads when people don't use them.
Why?
Because the customer wasn't the user - it was their boss and shareholders. It was all done to make someone else think 'woah, they are following the trend!'.
The core issue with generative AI is that it all works best when focused in a narrow sense. There is like one or two really clever uses I've seen - disappointingly, one of them was Jira. The internal jargon dictionary tool was legitimately impressive. Will it make any more money? Probably not.
I'm starting to get asked, "Could AI help you do such-and-such faster?" At first I tried to explain why the answer is no, because such-and-such doesn't lend itself to what AI is good at. But I'm starting to realize I'm going to have to tell them I am using it and maybe give them an example once in a while, because they're hearing too much about its wonderfulness to believe there's something it can't help with. They're going to think I'm just being stubborn even though I tell them I'm not opposed to using AI where it makes sense. If that means the job actually takes a little longer to add in the part where I use AI to speed it up, they'll be happier.
Previous management fads: https://en.wikipedia.org/wiki/Management_fad
Obviously in the right contexts, these methods provided value. But they became widely misapplied, causing a lot of harm.
And the Wikipedia list is far from exhaustive.
I realistically have between 10-100 questions I ask per day about things not immediately related to work. Double that if you include work based questions.
5% are succeeding. People are trying AI for just about everything right now. 5% is pretty damn good, when AI clearly has a lot of room to get better.
The good models are quite expensive and slow. The fast & cheap models aren't that great - unless very specifically fine-tuned.
Will it get better enough so that that growth rate in success pilots grows from 5% - 25% in 5 years or 20? Who knows, but it almost certainly will grow.
It's hard to tell how much better the top foundation models will get over the next 5-10 years, but one thing that's certain is that the cost will go down substantially for the same quality over that time frame.
Not to mention all the new use cases people will keep trying over that timeline.
If in 10-years time, AI is succeeding in 2x as many use cases - that might not justify current valuations, but it will be a much better future - and necessary if we're planning on having ~25% of the population being retired / not working by then.
Without AI replacing a lot of jobs, we're gonna have a tough time retiring all the people we promised retirements to.
About a mil? Maybe two? Seems realistic…
People have to invent whatever seems reasonable while squinting given how much accumulation of capital there is.
The guys with money are easy to fool. Just lie to them about your „product”, get the cash, get out of the rat race, smooth sailing.
Of course easier said than done. I can’t lie this convincingly, I don’t have the con man skillset or connections.
So I’m stuck in a 9 to 5. Zzz…
We use generative imagery/video at my job and it's adding value. I see value being added for coders.
There's real innovation happening, but I find it's mostly companies cutting corners making customer service even shittier than it already was.
> "Vaughan saw that his team was not fully on board. His ultimate response? He replaced nearly 80% of the staff within a year"
Being that this is Fortune magazine, it makes sense that they're portraying it this way, but reading between the lines there a little bit, it seems like the staff knew what would happen and wasn't keen on replacing themselves.
You could even say that comparing glorified ifs, and fors calculator to capabilities of even today's AIs is laughable.
The article mentions 19-20 year old founders, focused on solving single user problems, were the successes.
The sample size is 300 public AI deployments and an undisclosed number of private in-house AI projects. And the survey seems to only consider business applications, as compared with end-user applications like media and software. That's significant but not definitive.
Isn't it more likely that existing problems with low hanging fruit, perhaps unpopular answers, that could be solved by leaning on "AI". And perhaps "AI" wasn't the key to success?