LLMs put the free web at risk
Despite their claims, ChatGPT and Perplexity don’t provide live or complete information from the internet. Instead, they index only the parts of webpages their algorithms deem important. These limitations have given rise to a new breed of AI browser agents powered by the same foundational LLM tech.
These LLMs + agents will likely account for over 90% of web traffic in the near future. Bots scrape content without compensating websites for it. As humans access the web through LLMs and agents more, they directly view websites less. The loss of ad revenue from human visitors threatens many websites’ ability to provide their content publicly for free.
We’ve been considering solutions to these problems, and want to know what you think. 1) Are they problems? 2) How big and serious are they? 3) What are possible solutions?
We have some ideas…
21 comments
[ 2.2 ms ] story [ 63.8 ms ] threadTo paraphrase Upton Sinclair, “When your paycheck depends on you thinking something, it will be very hard to get you to not think it.”
Often now though before you see results, you see AI summaries. This is perfectly fine if someone is casually searching something like ratio of oil and vinegar for salad dressing. That person likely is making a salad right now and needs to know and does not care about reading through 5 screens of scrolling content, ads, popups, etc. to get a recipe.
Now if someone is researching the history of salad dressings and wants to go deep into types of oils, vinegars, mixture ratios, pairing with different lettuce types, fruits, nuts, etc. having content that provides that level of information is helpful, but to a smaller audience. That audience is more likely to click links to products or ads than someone just looking for a ratio right now.
The human impression counts for content sites is likely to go down, a lot for fluff content, but those sites are likely losing impressions from people that weren't clicking the ads or buying products anyhow.
I'm not in advertising/marketing so I don't know what impacts that has ... maybe for some time period, ad revenues will go down because sites lose the fluff visitors and number of visitors carries weight in advertising processes, right? Then maybe the processes catch up and realize that the good content sites have higher "stickyness" factor so conversion rates and length of users visit are much more important than page impression counts?
Maybe we don’t need to protect websites that rely on advertised content? Maybe it makes a better internet?
Based on what you’re saying here it sounds like good content will inevitably go behind paywalls in this case, and the number of paid subs people need to have will only continue to explode. Aka no free internet. Maybe that was bound to happen one way or another.
And all LLM providers will pay their data providers directly by API or something?
original internet (DARPA) was meant for communicating research and science. Maybe we return to that being the primary thing that will continue to be free?
https://www.goharvest.ai/
We recognize it as an unrewarding, tedious, and time-consuming thing humans have had to do until the latest abilities of browser agents.
As we built and learnt more about the industry we started to understand the underlying problems. For 99% of web sites web scraping isn’t the problem, the lack of compensation is.
We think there’s actually a better way to do this. If there’s enough demand, we can facilitate a rev share between agent scrapers and websites. Scrapers will pay less than what they pay for proxies and websites get a new revenue stream.
These are our thoughts at least so far. We aren’t ashamed of what we’ve built by any means in the way your comment implies lol. We want to see if we can benefit both parties in a win-win marketplace.
How is what you’re doing any better than what you are complaining about?
2) We aren’t complaining. We’re curious how others view this topic and space because it’s a contentious topic. We recognize that we might be able to address the larger issue of lack of compensation for websites being scraped by facilitating a win-win marketplace (only loser is proxy providers).
Are you paying any content providers now?
Why didn’t you just admit that up front or at least disclose you have a business interest in being able to scrape others content for free?
Remember what we all see now is the worst LLMs will ever be. They are only getting better now and that might even mean there will be a thousand different versions of a given article, each tailored for a give user bracket according to how compelling they predict the version will be to that specific cohort.
This is exactly right. I could imagine a bunch of horse trainers looking at early gas-powered automobiles in the late 1800's and thinking, "This 'automobile' thing isn't going to change much. Sure it's interesting, but they're so much slower and less reliable than a horse".
I have been paying $20/month for ChatGPT with web search just so I could avoid sites like yours.
The first thing I see when I go to your site is a half page cookie banner that wants me to accept tracking.
Of course I have an ad blocker. All of my bills are auto pay.
What site are you referring to? Ours doesn't have a cookie banner