Anthropic released 1,250 interviews about AI at work. Their headline: "predominantly positive sentiments." We ran the same interviews through structured LLM analysis, and the true story is a bit different.
Key findings:
• 85.7% have unresolved tensions (efficiency vs quality, convenience vs skill)
• Creatives struggle MOST yet adopt FASTEST
• Scientists have lowest anxiety despite lowest trust (see ai as a tool, plain and simple)
• 52% of creatives frame AI through "authenticity" (using it makes them feel like a fraud)
The story that’s solidifying is the tech is cool, it’s useful for certain things (eg, meeting note taking), but business have run a ton of “innovation lab” pilots that have returned little to no measurable value with leaders getting frustrated at the invested red ink. In short the substance isn't living up to the hype.
Everywhere I look the adoption metrics and impact metrics are a tiny fraction of what was projected/expected. Yes tech keynotes have their shiny examples of “success” but the data at scale tells a very different story and that’s increasingly hard to brush under the carpet.
Given the amount of financial engineering shenanigans and circular financing it’s unclear how much longer the present bonanza can continue before the financial and business reality playing out slams on the brakes.
The high usage and high anxiety tracks with what I have found from taking to artists IRL. There is a sense that any any public expression that is not wholly against AI will draw vilification from a section of the artistic community.
There are a broad range of opinions but the expression seems to have been extremely chilled.
```
Creatives have the highest struggle scores and the highest adoption rates.
```
Here is my guess for the puzzle: creative work is subjective and full of scaffolding. AI can easily generate this subjective scaffolding to a "good enough" level so it can get used without much scrutiny. This is very attractive for a creative to use on a day to day basis.
But, given the amount of content that wasn't created by the creative, the creative feels both a rejection of the work as foreign and a feeling of being replaced.
The path is less stark in more objective fields because the quality is objective, so harder to just accept a merely plausible solution, and the scaffolding is just scaffolding so who cares if it does the job.
I'm a scientist and I mostly agree with the scientist part, but I am definitely collaborating with my bot, I don't view it as "just a tool". I know this because this morning I had to do a forced reboot and my VsCode wasn't connecting to our remote servers, it took like over 5 minutes after reboot to reload my bot chat, and from like minutes 3-5 I had the distinct feeling of losing a valuable colleague.
This article is rife with unedited llm signals. This makes me question their methodology here. I want you believe what they found, but I don't trust this analysis. If they were this sloppy with the write up, how sloppy were they with the science?
Another thing I might throw out there is that there are so many domains and niches out there that person A and person B are almost certainly having genuinely different experiences with the same tools. So when person A says "wow this is the best thing ever" and person B says "this thing is horrible" they might both be right.
Is this any different than the adoption of any technology. I think of the transition from practical effects to CGI in Hollywood. Anxiety levels of the creative model builders was sky high at the time. It worked itself out and now there are different jobs.
I use AI coding almost daily. I’m able to move my repositories into context easily through the multitude of AI coding tools and I see a massive boost in productivity. I say this as a junior dev. Often the outputs are “almost” and I make the necessary fixes to get it the rest of the way there.
To contrast with this, my org tried using a simple QA bot for internal docs and has been struggled to move anything beyond proof of concept. The proof of concepts have been awful. It answers maybe 60-70% of questions correctly. The major issue seems to be related to taking PDFs laced with images and poorly written explanations. To get decent performance from these RAG bots, a large FAQ has to be written for every question it gets wrong. Of course this is just my org so it can’t necessarily be extrapolated across industry. However, how often have people come across a new team and find there is little to no documentation, poorly written documentation, or outdated documentation?
Where am I going with these two thoughts? Maybe the blocker to pushing more adoption within orgs is twofold, getting the correct context into the model and having decent context to start with.
Extracting value from these things is going to require a heavy lift in data curation and developing the harnesses. So far most of that effort has gone into coding. It will take time for the nontechnical and technical to work together to move the rest of an org into these tools in my opinion.
The big bet of course then is ROI and time to adoption vs current burn rates of the model providers.
What kind of mindset do you need to have to trust anything a company like this has to say? A company riding the hype train, praying the bubble doesn't pop, desperately trying to even turn a profit? Would you believe cigarettes are healthy too?
How do I know this fine article wasn’t the result of
“Create a web page infographic report that is convincing and boils down the essential truths of how people are feeling about AI in different professions and domains.. Include statistics and numbers and some rolling/animated sound bite quotes.”
13 comments
[ 2.9 ms ] story [ 32.4 ms ] threadEverywhere I look the adoption metrics and impact metrics are a tiny fraction of what was projected/expected. Yes tech keynotes have their shiny examples of “success” but the data at scale tells a very different story and that’s increasingly hard to brush under the carpet.
Given the amount of financial engineering shenanigans and circular financing it’s unclear how much longer the present bonanza can continue before the financial and business reality playing out slams on the brakes.
There are a broad range of opinions but the expression seems to have been extremely chilled.
Here is my guess for the puzzle: creative work is subjective and full of scaffolding. AI can easily generate this subjective scaffolding to a "good enough" level so it can get used without much scrutiny. This is very attractive for a creative to use on a day to day basis.
But, given the amount of content that wasn't created by the creative, the creative feels both a rejection of the work as foreign and a feeling of being replaced.
The path is less stark in more objective fields because the quality is objective, so harder to just accept a merely plausible solution, and the scaffolding is just scaffolding so who cares if it does the job.
To contrast with this, my org tried using a simple QA bot for internal docs and has been struggled to move anything beyond proof of concept. The proof of concepts have been awful. It answers maybe 60-70% of questions correctly. The major issue seems to be related to taking PDFs laced with images and poorly written explanations. To get decent performance from these RAG bots, a large FAQ has to be written for every question it gets wrong. Of course this is just my org so it can’t necessarily be extrapolated across industry. However, how often have people come across a new team and find there is little to no documentation, poorly written documentation, or outdated documentation?
Where am I going with these two thoughts? Maybe the blocker to pushing more adoption within orgs is twofold, getting the correct context into the model and having decent context to start with.
Extracting value from these things is going to require a heavy lift in data curation and developing the harnesses. So far most of that effort has gone into coding. It will take time for the nontechnical and technical to work together to move the rest of an org into these tools in my opinion.
The big bet of course then is ROI and time to adoption vs current burn rates of the model providers.
“Create a web page infographic report that is convincing and boils down the essential truths of how people are feeling about AI in different professions and domains.. Include statistics and numbers and some rolling/animated sound bite quotes.”
hell yeah