(For 2024-Q1, Reddit lost -$575 million on revenue of $242 M.)
If the quoted "$60 million deal"[1] from Feb 2024 is accurate, that small amount from Google may not be enough for Reddit to turn a profit. It remains to be seen what the Q2 or Q3 financials will show.
They were a public good currently larping as a for profit concern now run by a vanity and wealth driven executive driving it into the ground while it flails to monetize when that is likely incompatible with the entity.
Compare and contrast to say, HN, run on two servers in a colo with less than a handful of mods.
it's not just a message board, it's an influence machine.
They need to make sure the stuff they want people to think is posted often and has a big number next to it, they need to make sure things that people like are associated with the stuff they want people to like/think/do and things that people don't like are never associated with the stuff they want people to like/think/do. They need to make sure that people who say the wrong things are silenced or persuaded to leave, etc etc. Man they probably have at least one contact in at least one intelligence agency and they have to make sure not to run afoul of that contact.
Like the news isn't just a list of what happened recently, political debates aren't just two guys talking, and reddit/twitter aren't just message boards.
They spent 400M on R&D this quarter, which means more "personalisation"/ad targeting and probably cooking up some DOA chatbot/assistant product that's costing them a ton in compute
which is ironic because pre-AI every solid piece of obscure information and non-programming question usually had an answer on reddit, its an extremely valuable dataset looking back. but moving forward i think its only going to become less valuable and people will probably manually/custom-scrape all the questions out of worthwhile subreddits and open up their data for free
When I was young, my brother knew a guy who was really into movies. If you wanted to know about a movie you couldn't remember, you would go talk to that guy.
For a while, the internet had an end-run play that made that guy less useful. You can just go on the internet for obscure movie information, buddy.
But now it seems like knowing a movie guy is going to be the only way to get a real person's opinion on movies. The internet is about to forget everything without a profit motive and just start telling you that the latest product from a monolith corp like disney is the only movie worth watching. If someone scrapes all the useful movie opinions off of reddit and spends their time crafting it into a usable format, that guy's probably got a company. But not Bill. Bill's just a guy you can know or not know. You can't monetize knowing Bill. Sidenote that's probably why it irked me so bad when some bozo coined the phrase "social capital".
The API changes and these robots.txt were part of the same strategy - preventing third parties from scrapping their data and reducing the AI generated content that makes it into their data. So they can sell that data and make money.
# Welcome to Reddit's robots.txt
# Reddit believes in an open internet, but not the misuse of public content.
# See https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy Reddit's Public Content Policy for access and use restrictions to Reddit content.
# See https://www.reddit.com/r/reddit4researchers/ for details on how Reddit continues to support research and non-commercial use.
# policy: https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy
User-agent: *
Disallow: /
Independent scrapers can launder the data between Reddit and AI consumers. The only folks this hurts is users seeking info via search engines and folks willing to kowtow to rules that are potentially low impact to evade. Next steps would be (from an adversarial perspective) browser extensions that stream back data for ingestion similar to Recap for Pacer [1].
(full disclosure: assisting someone pursuing regulatory action against reddit in the EU for a separate issue from scraping, it's a valuable resource, but the folks who own and control it are meh)
Even more basic, it's free speech. The data itself is public domain so your free speech is not restricted and you don't need fair use excemptions for those restrictions. On the the access through the official system is restricted.
According to the US court systems the robots.txt file is meaningless. If they respond with a 200 status code giving you the access then you can legally scrape it all you want. If they require that you log in then you have to follow the terms you agree to when creating an account. Public means public though, and if Reddit doesn't want to make the content private (put it behind a login) then we can scrape away.
Note that scraping, regardless of the level of permission, doesn't mean you can do anything you want with the content. Copyright still applies. But you can scrape it, and if your use falls under Fair Use or another caveat to the copyright laws then you can do ahead and do it without needing any permission from the authors.
I liked the chapter on DMCA from the 5-volume E-Commerce &
Internet Law. It was super detailed.
I haven’t read volume 1, but apparently half of it is about data scraping, and I expect it to be similarly detailed. So if I were you, that’s where I’d start.
Another option is looking for “robots.txt” at Google Scholar and trying various keywords like “legality”, “scraping”, “case law”, etc.
Has not NYT tried to sue OpenAI because of them ignoring robots.txt or you mean it's impossible to prove and / or it's still more profitable to just ignore robots.txt?
That’s a weird statement to be absolutist about. The majority of individuals and companies who want to be successful do not do so by scrapping websites, thus have no reason to disobey robots.txt. Most people in the world, ambitious or not, wouldn’t even understand what your sentence refers to.
OP is obviously talking about people whose area of research/development/product would involve web scraping... This feels like being purposefully obtuse
it's even worse. it's not theirs (it's the users'), they are merely hosting it and using it (ToS gives them a fancy irrevocable license I guess).
so they can do whatever they want with it and the actual owners/authors have no chance to really influence Reddit at all to make it crawlable. (the GDPR-like data takeout is nice, but ... completely useless in these cases where the value is in the composition and aggregation with other users' content.)
I tried it many months back when they glitch-killed my decade-plus account. (Yes, I'm still bitter over the kafkaesque injustice.)
Anyway, you basically submit a request and then later they will email you a link to a zip file that contains a free dozen CSV files with unescaped newlines. One for all the comments you made, one for up/down-votes, one for blocked users, etc.
actually owners/authors like me would not want our stuff crawlable because that gives up our ownership.
When I am answering some random dude on reddit with a problem I want that dude to read my solution. I don't want this to be crawled and forever stored (probably deanonymized) or enshrined in a dozen commercial LLMs. There is substack for that stuff.
That's why more and more exchanges are moving to DMs and closed communities. Because we don't plan on stuff we intend for participants in the discussion to be harvested but it more and more is. If reddit breaks that trend I only welcome it.
I'll be honest... I don't care about this thing. I view public posts as something that belongs to the public domain and if that is searchable, all the better, other people can reach my post. And if that is what LLM's are trained on, also, great, I hope it will be useful to some people. What I _do_ care about is an even playing fields and access to _all_ llms to be trained in that data.
In the end, I view potential AGI's as a common consciousness brainchild.
OK. I see "AGI" as a buzzword and current ML trend as exploitation of creative works that profits big tech. They will always have bigger computers and get more out of what we create increasing wealth gap, unless we stop it
Social support (or we can even call it "[corporate] social responsibility") is a political question/problem/thing.
Endless iterations of the discussion about how copyright should/shouldn't be are meaningless without considering the larger social context.
The vast majority of creators (artists!?) are not compensated at all, and the users' (content consumers'? the public's?) attention is already 100% fully saturated.
So GenAI chrurning out more kitsch hardly changes that. (It got popular when it was novel, and ... that was it for now. And it became one more brush in the evergrowing workshop.)
And if/when a company creates a product out of it, then that product needs to be scrutinized and we should consider the ethical, social, and political problems. (Because ... politics is a blunt tool whereas ethics is as infinitely nuanced as we make it, whereas actual social considerations ought to be pragmatic and fair.)
And that's the problem. Users benefit from cheaper access to customized content. (In other words, arguably it's a net positive thing that they can - for example - ask some GenAI tool to make them a nice picture for their friend's birthday.) So what's the cost of this? Does it make some people jobless? Is that good or bad? (Is it good that plant breeding programs, fertilizers, tractors, and irrigation systems made a lot of agricultural workers jobless? Well, in some sense yes as allows a few people to feed many, freeing up time for others to become doctors and artists. In some sense bad, because our current socioeconomic system does not provide real social support - despite enormous redistribution of GDP. [Because brutal inefficiencies in allocation of that surplus. And that's again a political problem.])
... and of course here the status quo bias leads to "technological progress chipping away at inefficiencies" in a capitalist reality translates to "even more things get commoditized", and that in turn in the current shitty socioeconomic system equals "lots of externalities are not priced in, and lots of people are forced to drastically change their lives to adjust to new prices" (ie. price of their labor and/or products going down, so they need to change jobs, yet they barely get any support for doing so) ... and of course ethically it's bad that most people virtually uncritically accept and enjoy the results of progress (new products and services) without giving a fuck about the costs.
> Users benefit from cheaper access to customized content. (In other words, arguably it's a net positive thing that they can - for example - ask some GenAI tool to make them a nice picture for their friend's birthday.)
very arguably yeah. we evolve to where we push a button and get a birthday present. the logical conclusion is a system that does not even require to push a button. this is the opposite of the original point (spend time and effort making something to show you care). so to me it is not benefit, it is harm.
maybe you picked a bad example. but the examples of legit benefits of this tech, they don't turn into harms if you really analyze it and go to the root, are hard to find in consumer land.
I like this example because I tried making a greeting card "by hand" and it of course sucked ass. (As I have zero idea how to use Photoshop despite religiously reading the tutorials 20 years ago on a community site :D And, of course, I used PhotoPea.)
So by spending the same 20 minutes fiddling with some kind of graphics if the outcome becomes 10x better then I see that as an advantage. (That said I haven't tried this.)
Basically this is "exactly" that kind of capability that we already seen in "artist paints kid's drawing" [0] just commoditized. And this is where I (as a user) would be looking for the style transfer. Because I saw something famous/trendy/fancy/aesthetically-pleasing and now want to imitate it as a gag for this hypothetical birthday card. (Sure, copyright already doesn't care if I download someone's famous photo, and cut my friends head out from a photo I made, and put the head part on the copyrighted photo. Yet it was a straightforward derivative work. Just it's not commercial, damages are none, or even negative, ie. the artist benefits from that famous exposure! :D)
> legit benefits of this tech
I wanted to try to find an example that could apply to a lot of users. Because as awesome as using GenAI to generate Lean proofs (and then add a feedback loop by actually running the generated code through Lean) to solve Math Olympiad problems [1], it's not really an everyday thing.
> we evolve to where we push a button and get a birthday present. the logical conclusion is a system that does not even require to push a button.
Well, maybe! In no time we'll board the Axios and just chill. [2] But at the same time crafts and DIY and experiences (tourism, festivals, concerts) are ridiculously popular. (And it has its own problems. [3])
> Well, maybe! In no time we'll board the Axios and just chill.
This was sarcasm. That system would never happen. If birthday card requires no effort it is worse than nothing, literal noise.
> So by spending the same 20 minutes fiddling with some kind of graphics if the outcome becomes 10x better then I see that as an advantage. (That said I haven't tried this.)
Feels like you misunderstood. It is not "10x better" or "advantage" because literally the easier it was for you the less it is valuable.
> It is not "10x better" or "advantage" because literally the easier it was for you the less it is valuable.
... why? The closer I can approximate my imagined output (in a fixed time) the more value I see in it. The tool provides the added value.
We have better cameras than 30 years ago. Better pictures. It's easier to make a better photo in the same amount of time. I would be happier if my old blurry photos would be better.
the point it how difficult it is for you. the easier the present the less it is valuable. now you backtrack and mention that you have imagination and vision, and you spend effort trying to bring it to life, sure. but the original sentence was to just "ask some GenAI tool to make them a nice picture for their friend's birthday". which of course can be automated with a button.
You interpreted that part to mean "no customization", I meant as a tool for customization.
Value is relative, of course, that's why exchange/trade works, right? And the same applies here too, with the added complications of explicitly "unknowable price", after all I don't know apriori how much value my friend will assign to the card!
In general there are many situations that map closer to your understanding than mine, but I think you are dismissing also a lot of perfectly valid ones where your model simply doesn't apply.
For example if said friend values effort, but "apparent concept vs. actual results fidelity" is not part of effort for them.
Yet still we don't know if they value effort as in time spent (because if they value time spent then it's the same value in both of my scenarios) or if they value some "visible handcraftedness" (effort to produce something that I am known to be very amateur at). And even then the question arises, isn't picking the best tool for the job itself effort?
Of course if this friend simply abbors GenAI shit then they will think it's "low effort" even if I put hours and hours into it (that I wouldn't have put into a traditional image manipulation software, because there I realized soon that I would need months of learning, etc.)
Their license/Eula clearly state that Reddit has perpetual whatever to content posted on Reddit, but relying solely on DMCA for "stolen" content _yet again_ feels like a terrible way to deal with non-original content. Part of me hopes that Reddit gets hit with some new precidence-setting lawsuits regarding non-original content that requires useful attribution, but I double t that will ever happen.
An EULA does not change the morality of the situation anyway. They are a leech profiting off users generating content who are now upset about not getting a cut from third-parties also profiting from said user generated content.
can also say every humans are leech benefiting off free software (creators) and complaining about their worthless chitchat, barely usable because of its basic semantics, being "stolen".
1. That’s kind of the point of free software, is it not? If you don’t want people “leeching” off of your software for free, don’t make it free.
2. Reddit is an amazing source of coding information and general Q&A on an extremely wide variety of topics. I would not characterize all of it as chitchat.
With the amount of crap in Reddit, cleaning it must be a very non-trivial problem. (I mean, it never is, but in the case of Reddit it's probably extra complicated)
I mean, the reddit company did go public, so things like this were inevitable.
Also things like the API fiasco, and also small annoyances like the fact that when you click on an image on reddit, it now goes to a wrapper html page instead of just the actual image (this was one of the reasons reddit was better than most social media...).
Maybe it's just me or something temporary (I use Old Reddit, like all right-thinking folk) but for the past couple of days the image wrapper page seems to have been sent to the glue factory. I'm just getting the image now, unadorned.
It used to be that Reddit didn't host images and you'd have to link (often shitty) external image hosts. The someone created imgur to host images for reddit. And slowly but surely imgur became just another shitty image host (and social media site for some reason). Then reddit wanted some of the dough imgur was pulling in (probably just making losses) and added their own image hosting. At first it worked just like you are saying with you getting direct links to the image file. Now they also turned into yet another shitty image host.
Part of the blame for the redirect-to-wrapper page lies with browsers. If browsers didn't let servers reliably differentiated between a direct request and an <img> embed then this practice would not be as widespread.
While it's still not Reddit, but I've been enjoying Lemmy. I have a similar range of communities on each, and other than some annoying groupthink, the content is often similar.
And to me, forgetting to log in to each of them feels similar, too. For what that's worth. (I hate both of them when not logged in.)
I mostly contributed to r/nonsense and I'm pleased by the thought of that sub's content being used to train future AI, with information about the architectural uses of super-tall chef's hats, the prehistoric invasion of Europe by Beak People, and so forth.
My comment said, show me that the Google deal with Reddit is exclusive.
You haven't done that.
And there's no reason to think it would be, because of antitrust. The DOJ doesn't have to act "immediately", the point is that obvious antitrust violations come with fines that make it unprofitable to attempt in the first place. And this would be black-and-white obvious antitrust violation, given Google's monopoly status in search. This isn't a gray area where it might be worth it for Google to roll the dice.
clearly some deal was reach between the two parties or we wouldn't be here.
whether or not the deal is exclusive OR companies have to pay to index reddit it's still bad for competition. money has a barrier to entry preventing newcomers.
I can blame reddit for creating the deal and I can blame google for accepting the deal if the effect is bing, ddg and others cannot display reddit results without reaching some deal.
Blaming Google for accepting it makes no sense. That's like if a shopper goes to grocery store and buys an expensive $20 piece of cheese, and other shoppers can't afford cheese that pricey, and you're blaming that one shopper for buying it because it means other shoppers can't also get the cheese without paying for it. That doesn't make any sense. The store set the price, and they're the one to blame if other shoppers can't afford it.
If Bing, DDG and others can't reach a deal with Reddit, that has nothing to do with Google.
Again, blame here is 100% on Reddit, and 0% on Google. To assign blame to a purchaser in a case like this doesn't make any sense.
> bing probably has the money to reach a deal, smaller companies without monopolies is less likely, and that's the problem.
Reddit can charge smaller companies less money. So if there's a problem, again, the problem is 100% with Reddit.
Google is absolutely blameless here. You may not like Google, and you can certainly blame them for plenty of other things. But in this situation, literally all of the blame is with Reddit for deciding to remove their content from all search engines unless they pay. Reddit didn't have to do that. Google didn't make them do that.
Very few of the reddit users who are providing the content for free are motivated by which search engines are allowed to index the content, so I don't see how this would make it more ripe for competition. (If you just mean society would now be even better off if reddit were disrupted, ok, maybe, but that's a different thing.)
Remember, the only reason Reddit "won" was because Digg destroyed itself with a radical upgrade that everyone hated.
Reddit would have to do something similarly self-inflicted, and I can't even guess where people would go. Reddit was already an alternative to Digg -- what's the alternative to Reddit? I mean, it's certainly not Quora.
Reddit is quietly a huge website with a significant amount of users. So many people use it but dont talk about it. Google search says 1billion mau? Twice as big as Twitter
there are plenty of reddits outside of the "popular ones" that have invaluable information in one place and I can ask questions. It still has value for me, and not much value for axis of evil bots to spread political disinformation like popular subs and on twitter.
It was already dead by then. Really, it was the various Slashdot exoduses... sites like K5 got large initial boosts, but stumbled and started to deteriorate. If the Digg exodus is what sent you to Slashdot, chances are you're the kind of user everyone else was trying to escape.
>what's the alternative to Reddit? I mean, it's certainly not Quora.
If it was deliberate I certainly can't tell, but one of the characteristics of Reddit is that it caused so many other little tiny internet forums to just wither away. Most were visually unappealing, running some ancient phpbb software or whatever, but there were so many like stars in the night sky. Now, if they're even still running, you look for the newest post, and it will say "November 2023". Hell, the only reason they are still running is that the credit card number on file paying for hosting doesn't expire until next year somehow. Reddit is a red tide algae choking out all life in the ocean, nothing else gets to exist anymore.
>Reddit was already an alternative to Digg -- what's the alternative to Reddit?
This site is essentially 'orange reddit', they just need to add sub-HNs or tagging or something and it'd be ready for an influx of reddit refugees. Not that any of really want it, but it's possible.
The main thing I see Reddit being useful for are discussions about entertainment.
There's probably a subreddit for your favorite sports team, twitch steamer, TV show, book series video game, politics (which is entertainment for some people).
Reddit has seriously degraded the experience of a lot of these communities with things like restricting custom CSS.
It seems to me that the way you'd disrupt Reddit as a startup is to pick a vertical and laser focus on becoming the best discussion board for that community. If it's sports than have integrations for live stats, scores, etc.
In general you could attract users by offering profit sharing on ads the same way Youtube does for creators.
Have the best moderation tools in the world, a constant painpoint with Reddit. Give admins more flexibility over the appearance of the board, all things Reddit took away.
The other path for disruption would be if an established company with those communities tackled the problem. Lots of communities already us Discord, but they tend to also have a subreddit because chat and forums are different communication methods. Discord could easily offer a forum product as an extension of their chat services. If they do it well they'd drive a lot of users away from the subreddits.
Networks effects are more powerful than we are. Witness the number of people who despise Xhitter but are still on there. Once something has a sufficient network effect they become immune to normal market forces and able to abuse their position with near impunity.
The strange thing to me is how everybody keeps trying to make distributed Twitter happen when distributed Reddit is the low hanging fruit for federated social media.
You don't want to end up banned from a movies forum because you also participate in a political forum. Federation solves that problem because you can use separate accounts without either forum knowing that you also use the other.
Is this not just what the internet was before reddit? What features would "distributed reddit" have that an internet full of independent community forums be missing?
>The strange thing to me is how everybody keeps trying to make distributed Twitter happen when distributed Reddit is the low hanging fruit for federated social media.
Honestly, it's strange to me how hard people are trying to make distributed anything happen. Federation mostly solves a problem that real people don't have or care about.
>Honestly, it's strange to me how hard people are trying to make distributed anything happen.
IMO, something federation is very good at is solving one slow-moving problem - enshittification of social platforms. It's not immune, of course, but an Elon Musk-style takeover is much harder with Mastodon than Twitter, and it would be hard to run it into the ground in other ways because the platforms are owned by different people and groups.
It's not possible because the most common problems with running a forum are spam and moderation, and both of those are too much work unless it's centralized.
The killing of third party clients didn't have significant impact, I don't know what would they have to do to lose users, other than some kind of mandatory subscription fee.
It feels like Reddit is approaching an inflection point anyway where bot-made content is concentrated enough to spoil the whole experience. Closed servers like Discord and Slack may be the last haven of online human interaction.
How many other sites might have leverage to charge to be indexed?
I don't want to live in a world where you have to use X search engine to get answers from Y site - but this seems like the beginning of that world.
From an efficiency perspective - it's obviously better for websites to just lease their data to search engines then both sides paying tons of bandwidth and compute to get that data onto search engines.
Realistically, there are only 2 search engines now.
This seems very bad for Kagi - but possibly could lead the old, cool, hobbiest & un-monetized web being reinvented?
It's a very long article so understandable that you did not read on and learn about other search engines crawling beyond GBY. Still there are indeed very few that are crawling at web scale, and internationally. We are at 8 billion pages and totally independent [0], hence expressing our concerns to 404 media after being blanked by Reddit.
> did not read on and learn about other search engines crawling beyond GBY. Still there are indeed very few that are crawling at web scale, and internationally
That's helpful clarification.
In criticism of the article, you might agree that
none of them have their own results
is a fairly absolute statement. It signals: Final word on the matter; no nuance to follow.
> Kagi also uses Yandex index so it would be unusual that something exists in Yandex and not in Kagi results.
I've no reason to doubt this so I'll attribute it to which results get pushed higher in the rankings. Different entities hold power in the E & W hemispheres and that can influence search results to a degree.
Past that, there are some things I get at Yandex because Kagi seems incapable of searching for them. ex: Long strings like MD5 and larger hashes.
edit:30 seconds after posting I searched a hash and Kagi sent back a ton of results. As recently as last week I got zero results (diff hash).
That's the 2nd time in a week Kagi disproved a complaint of mine. I hope they continue to do so.
Brave has their own search engine, yandex I only use for reverse image search, baidu's interface is really clean and feels like old school google... but I don't speak chinese so I can't use it.
Brave doesn't have its own index of the full web, and it's even less useful than Yandex. And very likely buys some of it, according to what I've heard. So it falls into the last category.
if that is true then they are lying on their site where they claim: "Brave Search operates from a fully independent search index"
do you have any reference for your claim?
i use brave search and find it very useful. very rarely there is something i can't find, and when i run into that other search engines are not much better.
Given the piles of spammy shit on Google these days, I'm wondering if "doesn't have its own index of the full web" is actually a competitive advantage.
That seems like the business model for streaming. You subscribe to X provider to watch Y series. So, as for streaming, I suppose a pirate bay search engine will come up
Pirate Bay is probably not the most optimal analogy, more like Anna's Archive imho [1], individually offered by web property scrape runs compressed into a package, maybe served by torrents like this Academic Torrents site example [2].
Scraper engine->validation/processing/cleanup->object storage->index + torrent serving is rough pipeline sketch.
idk man i bet you five bucks and a handshake it's just going to play out like the existing startup grift.
There's an established player with institutional protections, then a scrappy upstart takes a bunch of VC money, converts it into runway, gives away the product for free, gradually replaces and becomes the standard, then puts out an s-1 document saying "we don't make money and we never have, want to invest?" and then they start to enjoy all the institutional protections. Or they don't. Either way you pay yourself handsomely from the runway money so who cares.
The upstart gets indexed and has an API, the established player doesn't.
The upstart is more easily found and modular but the institutional player can refuse to be indexed to own their data and they can block their API to prevent ai slop from getting in and dominating their content.
> but this seems like the beginning of that world.
It's not the beginning, it's mere continuation.
Walled gardens have existed since the AOL days. They deteriorate over time but it doesn't prevent companies from trying (each time, in bigger attempts).
Kagi gets part of their index from Google, per the article, so perhaps that's the reason Kagi still works. Wonder if Vlad and Kagi will do (or have done) the calculus to see if buying crawlability from Reddit itself is cheaper than buying results from Google for Reddit search.
They changed robots.txt a month or so ago. For the first 19 years of life, reddit had a very permissive robots.txt. We allowed all by default and then only restricted certain poorly behaved agents (and Bender's Shiny Metal Ass(tm))
But I can understand why they made the change they did. The data was being abused.
My guess is that this was an oversight -- that they will do an audit and reopen it for search engines after those engines agree not to use the data for training, because let's face it, reddit is a for profit business and they have to protect their income streams.
One (in this case, 2) company's incentive for profit should not take priority over the usability/well being of the internet as a whole, ever, and is exactly why we are where we are now. This is an absolutely terrible precedent.
This is a false dichotomy. You can have services, and not have them devolve into complete unusability in the name of profit. This isn’t sustainable either. The myopic pursuit of short term gains at the expense of the product will collapse at some point in the future, no matter how much you believe in this weird frog-boil internet we’ve inherited now.
> The myopic pursuit of short term gains at the expense of the product will collapse at some point in the future,
The myopic pursuit of short-term gains is the only playbook that works. Long-term business strategy is a gamble, and today's businesses have all learned that they'd rather make hay when the sun is shining than be remembered as a good business.
Twitter tried a long-term playbook to reverse their unprofitable sinkhole of a website. That ended up with them being undervalued and sold to the highest bidder.
> Twitter tried a long-term playbook to reverse their unprofitable sinkhole of a website.
From what I recall reading at the time, Twitter was finally becoming profitable before the sale (last two years? It’s hard to find a source now since every story since is about some shit show or other post sale).
> That ended up with them being undervalued and sold to the highest bidder.
You make it seem they were in dire straits and had to be sold for scraps, but that’s far from the case. They sold for more than their valuation to the only bidder because they understood what a good deal it was for them. They forced the buyer to not back out, after all.
Complete unusability is when ai tools clone the content and people stop visiting the original service and participating. I'll leave it up to them to defend blocking duck duck go for example, but blocking "AI" bots for an online community is a matter of survival at this point.
Alternatively, it's because the base platform has also devolved into unusability. Both Reddit and Twitter are in a position where their info is easily scraped, and their community is barely worth the advertising/paid-premium experience they demand from you. As both platforms continue to decline in quality, you might not even need to replace the original service. Both businesses appear intent on getting replaced.
We did. As in we, the Internet, existed for a long time without anyone making money and we paid for the privilege. Websites were built and hosted at owner's expense, for years, with no expectation that they be financially rewarded. Sure some would run donation drives, or work with sponsors relevant to the community in question, but a whole ton, mine included, just cost me a lot of money over many years.
Those websites were definitely technically inferior, as the march of progress is unavoidable, but web hosting is cheaper than it's ever been. A VPS that utterly blows away what mine was capable of in 2007 for nearly a hundred a month can now be had for about $10 per month. Yet everyone wants these monolith platforms, but even that wouldn't be the worst thing ever, except that every one of these platforms has a backend to support that we in the Old Internet never did: a C-suite's worth of executives and millions of shareholders, who for some reason have decided that reddit can't exist unless reddit makes them reams and reams of money.
I'd be very, very interested to see how much of, even what's probably the most massive one of all, Facebook, is non-essential busywork that could easily be shut down tomorrow with no adverse effects to the platform. Firstly the entire executive class, just, they don't do shit to make Facebook the product. In fact I'd argue their decisions almost universally have made it worse as a product very consistently for it's entire lifetime. Then, all the marketing people. There's just no goddamn reason to advertise Facebook (or reddit for that matter) the brand is so ubiquitous, if you actually found someone who'd never heard of it, I'd give you a large chunk of money. Add to that, if Facebook was doing a good job of being what it ostensibly is, then people immediately become the best advertising, because people want to hang with people in these digital spaces. Then get rid of the people working to make Facebook addictive with dark patterns. Then get rid of the entire targeted ad division, because it's gross and inhumane. Pare the company down to engineers who build the product, and if anything, expand the moderation team so they can actually ensure the safety of the platform, and of course the IT staff to back them. Now what does Facebook cost to operate?
As far as I'm concerned, this pearl-clutching about "well websites have to make money" is grossly, grossly overstated. Websites don't cost that much to run. A ton of money is being siphoned off by the MBA parasites playing games in Excel all day. A ton more is being wasted developing features that advertisers want and users hate. A ton more is being funneled into making products artificially addictive to vulnerable people, to exploit them, so let's just not do that. And of course, leadership, rewarding themselves with generous compensation packages they aren't even remotely able to justify. Now what does your website cost to maintain? Surely not nothing, and for websites of substantial size, it will still be high, but I'm willing to bet it's a hell, hell, hell of a lot less than it was before.
Part of the issue is that it isn’t just the web, but the inevitable american corporate shareholder model. Even businesses could be mom and pop ified and made way more popular overnight: quit raising prices and cutting corners and it would actually stand for itself like a massive $7 burrito. However the expectation is that shareholders get returns. Costs must be cut. Prices must be raised. Margins must be improved. It doesn’t matter if this eats the business alive, as shareholders are sufficiently leveraged. The whole system is incentivized to select for inferior quality and taking all the available money on the table.
My rant above and your response reminded me of all those tons of MMO games out there that are ancient, with a tiny playerbase, that remain profitable nonetheless simply because if you have a product that people like using, putting it into maintenance mode and doing the bare minimum to keep it running is a perfectly valid business strategy. The companies that buy these service games and run them effectively just buy completed money printers and keep them operating. It's not going to make anyone rich probably, but it's a perfectly valid and profitable way to go about things.
The silicon valley "grow at all costs, always evolve and innovate forever" model is so detached from the reality of most businesses in my experience.
>The companies that buy these service games and run them effectively just buy completed money printers and keep them operating.
I hadn't really thought about that topic in that way before. Really explains why some of those older MMOs have no desire to really make any improvements, the owners are happy to just keep them powered up and collect a check but have no incentive to invest in making them better.
I think the notion that sometimes things are just "done" is incredibly undervalued in our industry. Frankly I wish a ton of games I play would STOP updating.
>I think the notion that sometimes things are just "done" is incredibly undervalued in our industry.
I agree, but also the flip side is that things rapidly switch from 'done and working' to 'dead' pretty quickly if no one is willing to do minor maintenance.
Popular websites that allow user content to be uploaded or linked do cost that much to run, due to content moderation.
There might be a small (relatively) forum here and there that a few good moderators are willing to slave away at keeping clean, but you will never see a website that allows user content with as many users as Reddit/Youtube/Instagram/etc be cheap.
Although, due to AI, the cost to spam the small forums might be so small that even they might come into the crosshairs.
Although it is quite surprising that mainly text websites (Reddit, Twitter) are hard to run sustainably but video and image websites (YouTube, Instagram, TikTok) can because it is easier to sell ads against them.
I don't see how that would fall to different people in reddit's case. I'm sure reddit has some moderators on staff but the vast, vast majority of their moderation happens on the proverbial front lines, which is basically all volunteers. I would hope there's a dedicated abuse team at Reddit that are actually paid people whom the volunteers can kick the truly sick shit to so it can be properly dealt with, but given the corporate culture Reddit has shown over the years, I also wouldn't be awfully surprised if it's JUST down to the volunteers either.
I think it's the other way around, in that people don't like to hear how Reddit has become important due to the death of independent forums and the degree to which information has become concentrated on the site.
The death of independent forums has been greatly exaggerated.
Of all the forums I used to be active in, many are still active. The ones that died did so because the community died (i.e. they did not shift to Reddit and the like).
Reddit is great simply because it allowed anyone to create a community. No need to get a LAMP stack and deal with security vulnerabilities in your forum SW.
These days you have Lemmy and its ilk. Much higher barrier than the old LAMP stack, but also much superior to it. I do hope it takes off.
Person extensively quoted in the article here. They are welcome to reach out. But not a single person from any level did that, nor replied to my polite requests to explain and engage. We first contacted them in early June and by 13th June, I had escalated to Steve Huffman @spez.
An acquaintance investigating Reddit's moderation mechanization inquired how a major subreddit was moderated after an Associated Press post was auto removed by automod. They were banned from said sub. They inquired why they were banned, and they shared they would share any responses with a journalism org (to be transparent where any replies would be going, because they are going to a journalism org). They were muted by mods for 28 days and were "told off" in a very poor manner (per the screenshots I've seen) by the anonymous mod who replied to them. They were then banned from Reddit for 3 days after an appeal for "harassment"; when they requested more info about what was considered harassment, they were ignored. Ergo, inquiring as to how the mods of a major sub are automodding non-biased journalism sources (the AP, in this case) without any transparency appears to be considered harassment by Reddit. The interaction was submitted to the FTC through their complaint system to contribute towards their existing antitrust investigation of Reddit.
Shared because it is unlikely Reddit responds except when required by law, so I recommend engaging regulators (FTC, and DOJ at the bare minimum) and legislators (primarily those focused on Section 230 reforms) whenever possible with regards to this entity. They're the only folks worth escalating to, as Reddit's incentives are to gate content, keep ad buyers happy, and keep the user base in check while they struggle to break even, sharing as little information publicly as possible along the way [1] [2].
The blocks for MojeekBot, as Cloudflare verified and respectful bot for 20 years, started before the robots.txt file changes. We first noticed in early June.
We thought it was an oversight too at first. It usually is. Large publishers have blocked us when they have not considered the details, but then reinstated us when we got in touch and explained.
> But I can understand why they made the change they did. The data was being abused.
Depends how you see it - if you see it as 'their' data (legally true) or if you see it as user content (how their users would likely see it).
If you see it as 'user content', they are actually selling the data to be abused by one company, rather than stopping it being abused at all.
From a commercial 'lets sell user data and make a profit' perspective I get it, although does seem short-sighted to decide to effectively de-list yourself from alternative search engines (guess they just got enough cash to make it worth their while).
Is that actually true? Reddit may indeed have a license to use that data (derived from their ToS), but I very much doubt they actually own the copyright to it. If I write a comment on Reddit, then copy-paste it somewhere else, can Reddit sue me for copyright infringement?
They demand that right. That doesn't mean they actually have a right to use the content in ways that are not directly required for the operation of the website or that are otherwise surprising to the average user. Putting something in the TOS doesn't always make it a valid contract.
I personally feel that this kind of "exclusive search only by Google deal" should result in an anti-trust case against Google. This is the kind of abuse of monopoly power that caused anti-trust laws to be passed in the 1890s.
Usually, to trigger any kind of anti-trust law, you need to have massive market share. In this case, for example, Reddit almost certainly hasn't committed any antitrust violations, because they're a relatively minor player in their market.
Similarly, if you start a vacuum cleaner company, you can make whatever exclusive deals you want. But if you control 80% of the market for vacuum cleaners, then you might need to be more careful about leveraging your market share in unfair ways.
If a company is part of a robust, competitive market (like Reddit), it's usually wiser to let customers vote with their wallets, and leave the government out of it. If a company becomes massively dominant (like Google or TicketMaster), and if it starts pushing exclusive contracts, it's much harder for customers to switch away.
how was it being abused. You still clicked on the information and saw the reddit ads? Now they won't get any of that from "rival" sites to google. I guess they figured the 60 million was more than that ad revenue. Seems greedy but I don't think it's illegal like others are suggesting.
Ah so when reddit uses user content for monetization it's ok but when others do it then it isn't? Reddit may want that double standard but I think the only thing they are going to achieve with this stunt is more people ignoring robots.txt.
Its not strange to me. Every single time I've followed a Reddit link from search results, I've got a short and fairly useless conversation that doesn't help me at all. So I have never understood why people like it.
Obviously, people do see value in it, or they wouldn't keep saying so! I would happily exclude Reddit links from search results though.
I think Reddit lost that kind of authenticity a while ago. Advertisers know the "search:reddit.com <product>" trick, and when you look at the number of upvotes, it costs _pennies_ to get your product trending in the comments.
I don't search reddit for <product> though I search it for <highly technical issue with product> because reddit is the only place where real people discuss such issues and the solutions to them.
> where real people hang out and have real conversations
I don't consider the discussions there to be "real" in any meaningful way, thanks to the extensive moderation.
From what I've seen, there typically ends up being a small handful of moderator-enforced narratives that are deemed "acceptable" for a given subreddit, and any commenters deviating from those narratives get banned, or their comments end up as "[removed]" by "[deleted]", or the comments get obscured with the "comment score below threshold" notice.
It's generally some of the most one-sided and blandest discussion around. Given that there's often no meaningful back-and-forth involving differing perspectives of any sort, I'm not even sure if it should be considered "discussion". It's more like regurgitation and repetition.
I've found the situation to be particularly bad on the Canadian locale-specific subreddits, for example, but a enough of the tech-oriented ones I've seen seem to end up like that, too.
Yeah, but each sub to a greater or lesser degree, has its own hivemind you'll be run out of town (or possibly even banned) for challenging. And the average member of Reddit is quite willing to spout off confidently incorrect BS and downvote people into the ground who actually know what they're talking about.
Not exactly always a reliable source of info outside uncontroversial niche topics or places like /r/AskHistorians that actually moderate. And even there I've seen the occasional humdinger.
What use do you get out of a search engine if not searching for reddit and other forums? The rest of the internet has become a cesspool of useless AI generated crap.
FYI, Kagi lets you do this and personalize it as you desire. They even share aggregated stats※ about which domains users choose to block/lower. (Mine generally match these stats.)
I sincerely appreciate the diligence. I really did see Pinterest results over a longish span. I may well have only noted their presence and not their absence - skewing my perspective.
Overall, my experience is very positive. I'm on many PCs throughout the day and I miss Kagi when it isn't there.
Kagi doesn't provide it natively because of performance issues. They contributed to ublacklist to add kagi search. You can use it with regex and matching rules
> If you need advanced blocking features, such as specifying rules by match patterns (e.g., ://.example.com/*) or by regular expressions (e.g., /example\.(net|org)/), the uBlacklist browser extension includes support for Kagi Search.
Kagi lets you configure the search engine to deprioritize or even fully eliminate search results. They ride on the back of Google's indexing so -- if you ever change your mind -- you could bring reddit searches back.
>This is great. It means I won't see Reddit content popping up all over search results in other engines.
Honestly, that makes those other engines way less valuable because for many topics, telling the engine to specifically narrow the results down to reddit comments is the only way to get a decent answer to what you're looking for. I'd definitely support blocking Quora from everything though.
Interesting. I have long found Reddit to be the an excellent source of solutions to problems. Stack Overflow usually beats it for programming specific stuff, but for everything else usually the most helpful answer comes from Reddit. It's a real person, helping another real person with a real problem.
Just like every security feature in the physical and digital worlds, security just inconveniences honest people and the cost to bypass reduces the amount of people who try.
Eventually it becomes expensive to scrape reddit's data and most people will stop.
Proxy IPs are also known and typically blocked. In fact, you can't even browse reddit without logging in when connected to most proxies.
Many web scraping companies have loads of phones hooked up in a rack in order to use mobile IPs. Companies can't just block mobile IPs because their site would become unusable for several city blocks (mobile IPs often correspond to a specific cell tower). This is the face of modern web scraping: https://i.imgur.com/U2RXi5G.jpeg
Hopefully this paves the way for antitrust action, but I won't hold my breath.
Reddit's justification for this is profoundly wrong. Their "public content policy" is absurd doublespeak, and counter to everything the open internet is and hopes to be. You cannot simultaneously call yourself "open" and "public" while refusing access to automated clients. Every client is automated. They even go so far as to say that "crawling" (also known as "downloading") is an "abuse" and violates user privacy.
This is absurd, and not justified. I would love to see legislation that restricted server operators' ability to prohibit automated access in this way, but I suppose it will never happen. Some people in this thread have attempted to justify the policy by saying "they have to protect their income streams". No they don't. You don't have a right to an income stream, and you certainly don't have a right to lie in order to get all the benefits of an open internet with none of the downsides. Noting of course that the "downsides" are in this case actually just "competitors".
not that reddit has a monopoly, but that google has.
google is using their power to prevent others from competing.
the problem here is of course that if reddit would be in financial trouble (i don't know if they are but let's imagine they need this money), they'd be between a rock and a hard place.
google should not be allowed to make exclusive deals, and reddit could not survive without the deal, then what would be left? google buys reddit, or the relevant authority approves of the deal?
i thought about the same problem with firefox. let's assume firefox is forced to allow people to make a choice of the default search engine (just like microsoft was forced to allow a choice of default browser on windows) then google might stop paying mozilla, and they could end up in financial trouble.
ideally no company ever depends on a single other company that much, but that only works if we don't allow companies to grow that much in the first place.
> let's assume firefox is forced to allow people to make a choice of the default search engine
Firefox has always allowed people to make a choice of the default search engine, since before it was even called Firefox. I know. I was there building it.
yes, but the default is google, and you have to go into the settings to make a choice, so most people keep the default. what i meant was the EU directive for microsoft where they actually had to put up a prompt at first use asking the user which browser they want, without allowing any default (and, i am not sure, maybe even a randomized list)
if the same was done for search engine choice for firefox then google would no longer be the default, and they would have no reason to pay firefox for that.
This is not true. Firefox comes with a fixed default on installation. You can change the search engine afterwards but that does not make the default a user choice.
Yes, sorry, should have been more clear: I claim google is in a monopoly position, not reddit. The rest of the comment is unrelated ranting about reddit's betrayal of their previously-held "public data is public" position.
Up until 2016 (I think, +/- 1 year), if you could remember 3 uncommon words in a comment, you could find any reddit post instantly on Google. I'd want to follow up on a thread from weeks ago, and it was magic. Number one result. Then one day that just stopped working, and even adding site:*.reddit.com didn't fix it. At the time, I think, I didn't realize that it was mostly Google's fault, I thought maybe Reddit had changed their infrastructure so that it couldn't be crawled properly.
Google hasn't been a search engine in a long while, it's just an advertisement engine now.
it's so bad it's crazy, you can legit not find stuff on the internet anymore, it's the same with youtube, I search something and get like 20 or so results and then everything else is hidden.
it started when youtube removed the ability to search for videos older than 5 years, if I had to guess? cost saving, have every old video in cheaper storage... but it sort of fragments youtube, every couple of years you only get newer content.
One day I was searching live videos of a local band from youtube and when sorting by upload date the oldest video was from 2010. I knew for a fact there had to be older videos so I got a youtube API key and searched via the API, ended up finding multiple videos starting from 2006. Learned that youtube is full of videos that are basically impossible to find with the regular search.
One of my past times is looking up anime opening reactions for fun/to hear people listen to bands I enjoy that sometimes do anime ops, searching that is so scary, you see the same 4 people or every month or so you get 1 or get 1 removed, you can't tell me there's not more than 4 people on the internet that upload opening reactions... when anime is a billion+ dollar industry, if you know particular channels you can see daily uploads on plenty of channels, that are simply search banned for whatever reason by the algorithm.
But yeah, the most outrageous one is older videos, I do believe the reason is that they are using some long term cloud storage that is cheaper for older videos so they removed the ability to search by date.
Additionally, I don't believe the API fully fixes it, because Bing has a wrapper for youtube and searches do not really vary
"we noticed that since our search results had gotten so bad nobody can use them to find the things they want, people just kept adding "reddit" to search terms anyways, so we figured we might as well make it official and exclusive"
378 comments
[ 4.4 ms ] story [ 344 ms ] threadThe most recent 10-K financial results 2024-03-31 (filed 2024-05-08) shows they actually lost money: https://www.sec.gov/edgar/browse/?CIK=1713445
(For 2024-Q1, Reddit lost -$575 million on revenue of $242 M.)
If the quoted "$60 million deal"[1] from Feb 2024 is accurate, that small amount from Google may not be enough for Reddit to turn a profit. It remains to be seen what the Q2 or Q3 financials will show.
[1] https://www.google.com/search?q=google+ai+deal+reddit
I just read they have 2000 employees which is also puzzling to me
Compare and contrast to say, HN, run on two servers in a colo with less than a handful of mods.
They need to make sure the stuff they want people to think is posted often and has a big number next to it, they need to make sure things that people like are associated with the stuff they want people to like/think/do and things that people don't like are never associated with the stuff they want people to like/think/do. They need to make sure that people who say the wrong things are silenced or persuaded to leave, etc etc. Man they probably have at least one contact in at least one intelligence agency and they have to make sure not to run afoul of that contact.
Like the news isn't just a list of what happened recently, political debates aren't just two guys talking, and reddit/twitter aren't just message boards.
Startpage, Kagi and Lukol are 3 that source from Google. I imagine there are others.
For a while, the internet had an end-run play that made that guy less useful. You can just go on the internet for obscure movie information, buddy.
But now it seems like knowing a movie guy is going to be the only way to get a real person's opinion on movies. The internet is about to forget everything without a profit motive and just start telling you that the latest product from a monolith corp like disney is the only movie worth watching. If someone scrapes all the useful movie opinions off of reddit and spends their time crafting it into a usable format, that guy's probably got a company. But not Bill. Bill's just a guy you can know or not know. You can't monetize knowing Bill. Sidenote that's probably why it irked me so bad when some bozo coined the phrase "social capital".
Then again most of what that site does is just blend and regurgitate the information that's currently on it anyway.
Love how it's their data when it might make them money but not their data if they get sued.
https://old.reddit.com/r/redditdev/comments/1doc3pt/updating...
[1] https://free.law/recap/faq
(full disclosure: assisting someone pursuing regulatory action against reddit in the EU for a separate issue from scraping, it's a valuable resource, but the folks who own and control it are meh)
Note that scraping, regardless of the level of permission, doesn't mean you can do anything you want with the content. Copyright still applies. But you can scrape it, and if your use falls under Fair Use or another caveat to the copyright laws then you can do ahead and do it without needing any permission from the authors.
I haven’t read volume 1, but apparently half of it is about data scraping, and I expect it to be similarly detailed. So if I were you, that’s where I’d start.
Another option is looking for “robots.txt” at Google Scholar and trying various keywords like “legality”, “scraping”, “case law”, etc.
Of course, that became unsustainable so now I have everything behind a login wall.
You can see it here: https://search.google.com/test/rich-results/result?id=_mYogl... (click on "View Tested Page")
Calling it "public" content in the very act of exercising their ownership over it. The balls on whoever wrote that.
so they can do whatever they want with it and the actual owners/authors have no chance to really influence Reddit at all to make it crawlable. (the GDPR-like data takeout is nice, but ... completely useless in these cases where the value is in the composition and aggregation with other users' content.)
Is there a way to export my history? How?
(and there's some help article for it that I didn't read, but google found this first https://support.reddithelp.com/hc/en-us/articles/36004304835... that's how I got to the link)
Anyway, you basically submit a request and then later they will email you a link to a zip file that contains a free dozen CSV files with unescaped newlines. One for all the comments you made, one for up/down-votes, one for blocked users, etc.
When I am answering some random dude on reddit with a problem I want that dude to read my solution. I don't want this to be crawled and forever stored (probably deanonymized) or enshrined in a dozen commercial LLMs. There is substack for that stuff.
In the end, I view potential AGI's as a common consciousness brainchild.
Endless iterations of the discussion about how copyright should/shouldn't be are meaningless without considering the larger social context.
The vast majority of creators (artists!?) are not compensated at all, and the users' (content consumers'? the public's?) attention is already 100% fully saturated.
So GenAI chrurning out more kitsch hardly changes that. (It got popular when it was novel, and ... that was it for now. And it became one more brush in the evergrowing workshop.)
And if/when a company creates a product out of it, then that product needs to be scrutinized and we should consider the ethical, social, and political problems. (Because ... politics is a blunt tool whereas ethics is as infinitely nuanced as we make it, whereas actual social considerations ought to be pragmatic and fair.)
And that's the problem. Users benefit from cheaper access to customized content. (In other words, arguably it's a net positive thing that they can - for example - ask some GenAI tool to make them a nice picture for their friend's birthday.) So what's the cost of this? Does it make some people jobless? Is that good or bad? (Is it good that plant breeding programs, fertilizers, tractors, and irrigation systems made a lot of agricultural workers jobless? Well, in some sense yes as allows a few people to feed many, freeing up time for others to become doctors and artists. In some sense bad, because our current socioeconomic system does not provide real social support - despite enormous redistribution of GDP. [Because brutal inefficiencies in allocation of that surplus. And that's again a political problem.])
... and of course here the status quo bias leads to "technological progress chipping away at inefficiencies" in a capitalist reality translates to "even more things get commoditized", and that in turn in the current shitty socioeconomic system equals "lots of externalities are not priced in, and lots of people are forced to drastically change their lives to adjust to new prices" (ie. price of their labor and/or products going down, so they need to change jobs, yet they barely get any support for doing so) ... and of course ethically it's bad that most people virtually uncritically accept and enjoy the results of progress (new products and services) without giving a fuck about the costs.
very arguably yeah. we evolve to where we push a button and get a birthday present. the logical conclusion is a system that does not even require to push a button. this is the opposite of the original point (spend time and effort making something to show you care). so to me it is not benefit, it is harm.
maybe you picked a bad example. but the examples of legit benefits of this tech, they don't turn into harms if you really analyze it and go to the root, are hard to find in consumer land.
So by spending the same 20 minutes fiddling with some kind of graphics if the outcome becomes 10x better then I see that as an advantage. (That said I haven't tried this.)
Basically this is "exactly" that kind of capability that we already seen in "artist paints kid's drawing" [0] just commoditized. And this is where I (as a user) would be looking for the style transfer. Because I saw something famous/trendy/fancy/aesthetically-pleasing and now want to imitate it as a gag for this hypothetical birthday card. (Sure, copyright already doesn't care if I download someone's famous photo, and cut my friends head out from a photo I made, and put the head part on the copyrighted photo. Yet it was a straightforward derivative work. Just it's not commercial, damages are none, or even negative, ie. the artist benefits from that famous exposure! :D)
> legit benefits of this tech
I wanted to try to find an example that could apply to a lot of users. Because as awesome as using GenAI to generate Lean proofs (and then add a feedback loop by actually running the generated code through Lean) to solve Math Olympiad problems [1], it's not really an everyday thing.
> we evolve to where we push a button and get a birthday present. the logical conclusion is a system that does not even require to push a button.
Well, maybe! In no time we'll board the Axios and just chill. [2] But at the same time crafts and DIY and experiences (tourism, festivals, concerts) are ridiculously popular. (And it has its own problems. [3])
[0] https://www.youtube.com/watch?v=dB-Q0eNsUaQ KID'S ART Redrawn by a PROFESSIONAL ARTIST! - Ep.6
[1] https://news.ycombinator.com/item?id=41069829
[2] https://www.thelist.com/img/gallery/things-only-adults-notic...
[3] https://edition.cnn.com/2024/07/08/travel/barcelona-tourism-...
This was sarcasm. That system would never happen. If birthday card requires no effort it is worse than nothing, literal noise.
> So by spending the same 20 minutes fiddling with some kind of graphics if the outcome becomes 10x better then I see that as an advantage. (That said I haven't tried this.)
Feels like you misunderstood. It is not "10x better" or "advantage" because literally the easier it was for you the less it is valuable.
... why? The closer I can approximate my imagined output (in a fixed time) the more value I see in it. The tool provides the added value.
We have better cameras than 30 years ago. Better pictures. It's easier to make a better photo in the same amount of time. I would be happier if my old blurry photos would be better.
Value is relative, of course, that's why exchange/trade works, right? And the same applies here too, with the added complications of explicitly "unknowable price", after all I don't know apriori how much value my friend will assign to the card!
In general there are many situations that map closer to your understanding than mine, but I think you are dismissing also a lot of perfectly valid ones where your model simply doesn't apply.
For example if said friend values effort, but "apparent concept vs. actual results fidelity" is not part of effort for them.
Yet still we don't know if they value effort as in time spent (because if they value time spent then it's the same value in both of my scenarios) or if they value some "visible handcraftedness" (effort to produce something that I am known to be very amateur at). And even then the question arises, isn't picking the best tool for the job itself effort?
Of course if this friend simply abbors GenAI shit then they will think it's "low effort" even if I put hours and hours into it (that I wouldn't have put into a traditional image manipulation software, because there I realized soon that I would need months of learning, etc.)
can also say every humans are leech benefiting off free software (creators) and complaining about their worthless chitchat, barely usable because of its basic semantics, being "stolen".
I will never take a statement given by a company that blatantly lies like this at face value going forward. What a bunch of clowns.
Also things like the API fiasco, and also small annoyances like the fact that when you click on an image on reddit, it now goes to a wrapper html page instead of just the actual image (this was one of the reasons reddit was better than most social media...).
Part of the blame for the redirect-to-wrapper page lies with browsers. If browsers didn't let servers reliably differentiated between a direct request and an <img> embed then this practice would not be as widespread.
And to me, forgetting to log in to each of them feels similar, too. For what that's worth. (I hate both of them when not logged in.)
Blame Reddit, not Google.
Is Google's contract with Reddit exclusive, so that other search engines aren't given the opportunity to also pay?
I highly doubt that, especially since the DOJ would go after that immediately because of antitrust.
So no, pretty sure the blame here is 100% on Reddit unless you have evidence otherwise.
> so that other search engines aren't given the opportunity to also pay?
this makes it harder for new engines if google has exclusive deals with some of the most popular sites
My comment said, show me that the Google deal with Reddit is exclusive.
You haven't done that.
And there's no reason to think it would be, because of antitrust. The DOJ doesn't have to act "immediately", the point is that obvious antitrust violations come with fines that make it unprofitable to attempt in the first place. And this would be black-and-white obvious antitrust violation, given Google's monopoly status in search. This isn't a gray area where it might be worth it for Google to roll the dice.
whether or not the deal is exclusive OR companies have to pay to index reddit it's still bad for competition. money has a barrier to entry preventing newcomers.
I can blame reddit for creating the deal and I can blame google for accepting the deal if the effect is bing, ddg and others cannot display reddit results without reaching some deal.
I'm saying the blame is 100% with Reddit.
Blaming Google for accepting it makes no sense. That's like if a shopper goes to grocery store and buys an expensive $20 piece of cheese, and other shoppers can't afford cheese that pricey, and you're blaming that one shopper for buying it because it means other shoppers can't also get the cheese without paying for it. That doesn't make any sense. The store set the price, and they're the one to blame if other shoppers can't afford it.
If Bing, DDG and others can't reach a deal with Reddit, that has nothing to do with Google.
Again, blame here is 100% on Reddit, and 0% on Google. To assign blame to a purchaser in a case like this doesn't make any sense.
i don't think google is blameless like you propose.
Reddit can charge smaller companies less money. So if there's a problem, again, the problem is 100% with Reddit.
Google is absolutely blameless here. You may not like Google, and you can certainly blame them for plenty of other things. But in this situation, literally all of the blame is with Reddit for deciding to remove their content from all search engines unless they pay. Reddit didn't have to do that. Google didn't make them do that.
Reddit did this. Not Google.
They could monetize it much better while being less annoying.
Ultimately - Google is getting everything they want from Reddit with this deal without having to buy it outright.
Short of Reddit transforming to an entirely different product (difficult) - I'm not sure where the major growth opportunity is for it.
It'd be somewhat hilarious if google bought reddit just to archive it and shut it down.
Remember, the only reason Reddit "won" was because Digg destroyed itself with a radical upgrade that everyone hated.
Reddit would have to do something similarly self-inflicted, and I can't even guess where people would go. Reddit was already an alternative to Digg -- what's the alternative to Reddit? I mean, it's certainly not Quora.
>what's the alternative to Reddit? I mean, it's certainly not Quora.
If it was deliberate I certainly can't tell, but one of the characteristics of Reddit is that it caused so many other little tiny internet forums to just wither away. Most were visually unappealing, running some ancient phpbb software or whatever, but there were so many like stars in the night sky. Now, if they're even still running, you look for the newest post, and it will say "November 2023". Hell, the only reason they are still running is that the credit card number on file paying for hosting doesn't expire until next year somehow. Reddit is a red tide algae choking out all life in the ocean, nothing else gets to exist anymore.
It's like what happened to personal websites when things like Blogger, Tumblr and Facebook popped up.
It's hard to beat something that is easy to set up and pays for hosting but still let's you control moderation. It's a no brainer.
Managing your own domain where users post content is a minefield of problems these days even if you didn't mind the cost of running it.
Something like this might also explain the move to things like Discord over IRC.
This site is essentially 'orange reddit', they just need to add sub-HNs or tagging or something and it'd be ready for an influx of reddit refugees. Not that any of really want it, but it's possible.
The main thing I see Reddit being useful for are discussions about entertainment.
There's probably a subreddit for your favorite sports team, twitch steamer, TV show, book series video game, politics (which is entertainment for some people).
Reddit has seriously degraded the experience of a lot of these communities with things like restricting custom CSS.
It seems to me that the way you'd disrupt Reddit as a startup is to pick a vertical and laser focus on becoming the best discussion board for that community. If it's sports than have integrations for live stats, scores, etc.
In general you could attract users by offering profit sharing on ads the same way Youtube does for creators.
Have the best moderation tools in the world, a constant painpoint with Reddit. Give admins more flexibility over the appearance of the board, all things Reddit took away.
The other path for disruption would be if an established company with those communities tackled the problem. Lots of communities already us Discord, but they tend to also have a subreddit because chat and forums are different communication methods. Discord could easily offer a forum product as an extension of their chat services. If they do it well they'd drive a lot of users away from the subreddits.
You don't want to end up banned from a movies forum because you also participate in a political forum. Federation solves that problem because you can use separate accounts without either forum knowing that you also use the other.
I would like to see a wikipedia-style system for Twitter/Reddit: open access data, non-profit.
Honestly, it's strange to me how hard people are trying to make distributed anything happen. Federation mostly solves a problem that real people don't have or care about.
IMO, something federation is very good at is solving one slow-moving problem - enshittification of social platforms. It's not immune, of course, but an Elon Musk-style takeover is much harder with Mastodon than Twitter, and it would be hard to run it into the ground in other ways because the platforms are owned by different people and groups.
Does this mean there will be a future where everyone is running their own crawler? I suppose.
How many other sites might have leverage to charge to be indexed?
I don't want to live in a world where you have to use X search engine to get answers from Y site - but this seems like the beginning of that world.
From an efficiency perspective - it's obviously better for websites to just lease their data to search engines then both sides paying tons of bandwidth and compute to get that data onto search engines.
Realistically, there are only 2 search engines now.
This seems very bad for Kagi - but possibly could lead the old, cool, hobbiest & un-monetized web being reinvented?
edit:
> Realistically, there are only 2 search engines now.
https://seirdy.one/posts/2021/03/10/search-engines-with-own-...
From the article:
This seems to assert that ~0 other search providers do any crawling at all. Ever. Are we sure that's the case?[0] https://www.mojeek.com/about/why-mojeek
That's helpful clarification.
In criticism of the article, you might agree that
none of them have their own results
is a fairly absolute statement. It signals: Final word on the matter; no nuance to follow.
Except that isn't what happened. Within the context, 'almost none' referred to size of the group, not to the amount of crawling.
I was discussing how much crawling is being done - outside of the 'almost none' sized group.
The 'almost none' sized group has 'their own results', so they do crawl. Based on that, the rest do not. Ergo, they do not crawl.
A search engine that never crawls seems non-intuitive.
I'm not taking their reporting without compensation, but that also means I didn't have the whole story. Such is life in this era of the internet.
And Yandex isn't much better for non cyrillic search, Baidu is only for the Chinese web effectively.
And all other search engines either don't even attempt to do full web crawls anymore and/or buy from one of the four above.
So realistically there's just one search engine for the full web that actually does the work.
I like Yandex when I'm rabbit-holing after obscure musicians/music. I routinely have a better experience than I do with DDG or Kagi or Goog.
I've no reason to doubt this so I'll attribute it to which results get pushed higher in the rankings. Different entities hold power in the E & W hemispheres and that can influence search results to a degree.
Past that, there are some things I get at Yandex because Kagi seems incapable of searching for them. ex: Long strings like MD5 and larger hashes.
edit:30 seconds after posting I searched a hash and Kagi sent back a ton of results. As recently as last week I got zero results (diff hash).
That's the 2nd time in a week Kagi disproved a complaint of mine. I hope they continue to do so.
I hope that one day they get a western version
do you have any reference for your claim?
i use brave search and find it very useful. very rarely there is something i can't find, and when i run into that other search engines are not much better.
Instead the wording leaves wiggle room for the possibility of using multiple.
https://www.crawlson.com/ https://search.marginalia.nu/ https://wiby.me/ https://searchmysite.net/
Scraper engine->validation/processing/cleanup->object storage->index + torrent serving is rough pipeline sketch.
[1] https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... ("HN Search: annas archive")
[2] https://academictorrents.com/details/9c263fc85366c1ef8f5bb9d... ("AcademicTorrents: Reddit comments/submissions 2005-06 to 2023-12 [2.52TB]")
There's an established player with institutional protections, then a scrappy upstart takes a bunch of VC money, converts it into runway, gives away the product for free, gradually replaces and becomes the standard, then puts out an s-1 document saying "we don't make money and we never have, want to invest?" and then they start to enjoy all the institutional protections. Or they don't. Either way you pay yourself handsomely from the runway money so who cares.
The upstart gets indexed and has an API, the established player doesn't.
The upstart is more easily found and modular but the institutional player can refuse to be indexed to own their data and they can block their API to prevent ai slop from getting in and dominating their content.
It's not the beginning, it's mere continuation.
Walled gardens have existed since the AOL days. They deteriorate over time but it doesn't prevent companies from trying (each time, in bigger attempts).
It still exists. It just isn't that popular.
https://help.kagi.com/kagi/search-details/search-sources.htm...
>Our search results also include anonymized API calls to all major search result providers worldwide
Incorrect: https://www.mojeek.com/about/why-mojeek
I assumed he had financial connection to them, but didn't want to take the time to research it. Mojeek is the new fetch.
> Searching for Reddit still works on Kagi, an independent, paid search engine that buys part of its search index from Google.
But I can understand why they made the change they did. The data was being abused.
My guess is that this was an oversight -- that they will do an audit and reopen it for search engines after those engines agree not to use the data for training, because let's face it, reddit is a for profit business and they have to protect their income streams.
The myopic pursuit of short-term gains is the only playbook that works. Long-term business strategy is a gamble, and today's businesses have all learned that they'd rather make hay when the sun is shining than be remembered as a good business.
Twitter tried a long-term playbook to reverse their unprofitable sinkhole of a website. That ended up with them being undervalued and sold to the highest bidder.
From what I recall reading at the time, Twitter was finally becoming profitable before the sale (last two years? It’s hard to find a source now since every story since is about some shit show or other post sale).
> That ended up with them being undervalued and sold to the highest bidder.
You make it seem they were in dire straits and had to be sold for scraps, but that’s far from the case. They sold for more than their valuation to the only bidder because they understood what a good deal it was for them. They forced the buyer to not back out, after all.
Those websites were definitely technically inferior, as the march of progress is unavoidable, but web hosting is cheaper than it's ever been. A VPS that utterly blows away what mine was capable of in 2007 for nearly a hundred a month can now be had for about $10 per month. Yet everyone wants these monolith platforms, but even that wouldn't be the worst thing ever, except that every one of these platforms has a backend to support that we in the Old Internet never did: a C-suite's worth of executives and millions of shareholders, who for some reason have decided that reddit can't exist unless reddit makes them reams and reams of money.
I'd be very, very interested to see how much of, even what's probably the most massive one of all, Facebook, is non-essential busywork that could easily be shut down tomorrow with no adverse effects to the platform. Firstly the entire executive class, just, they don't do shit to make Facebook the product. In fact I'd argue their decisions almost universally have made it worse as a product very consistently for it's entire lifetime. Then, all the marketing people. There's just no goddamn reason to advertise Facebook (or reddit for that matter) the brand is so ubiquitous, if you actually found someone who'd never heard of it, I'd give you a large chunk of money. Add to that, if Facebook was doing a good job of being what it ostensibly is, then people immediately become the best advertising, because people want to hang with people in these digital spaces. Then get rid of the people working to make Facebook addictive with dark patterns. Then get rid of the entire targeted ad division, because it's gross and inhumane. Pare the company down to engineers who build the product, and if anything, expand the moderation team so they can actually ensure the safety of the platform, and of course the IT staff to back them. Now what does Facebook cost to operate?
As far as I'm concerned, this pearl-clutching about "well websites have to make money" is grossly, grossly overstated. Websites don't cost that much to run. A ton of money is being siphoned off by the MBA parasites playing games in Excel all day. A ton more is being wasted developing features that advertisers want and users hate. A ton more is being funneled into making products artificially addictive to vulnerable people, to exploit them, so let's just not do that. And of course, leadership, rewarding themselves with generous compensation packages they aren't even remotely able to justify. Now what does your website cost to maintain? Surely not nothing, and for websites of substantial size, it will still be high, but I'm willing to bet it's a hell, hell, hell of a lot less than it was before.
The silicon valley "grow at all costs, always evolve and innovate forever" model is so detached from the reality of most businesses in my experience.
I hadn't really thought about that topic in that way before. Really explains why some of those older MMOs have no desire to really make any improvements, the owners are happy to just keep them powered up and collect a check but have no incentive to invest in making them better.
I agree, but also the flip side is that things rapidly switch from 'done and working' to 'dead' pretty quickly if no one is willing to do minor maintenance.
Popular websites that allow user content to be uploaded or linked do cost that much to run, due to content moderation.
There might be a small (relatively) forum here and there that a few good moderators are willing to slave away at keeping clean, but you will never see a website that allows user content with as many users as Reddit/Youtube/Instagram/etc be cheap.
Although, due to AI, the cost to spam the small forums might be so small that even they might come into the crosshairs.
Reddit outsourced most of it's moderation to unpaid volunteers.
Of all the forums I used to be active in, many are still active. The ones that died did so because the community died (i.e. they did not shift to Reddit and the like).
Reddit is great simply because it allowed anyone to create a community. No need to get a LAMP stack and deal with security vulnerabilities in your forum SW.
These days you have Lemmy and its ilk. Much higher barrier than the old LAMP stack, but also much superior to it. I do hope it takes off.
Shared because it is unlikely Reddit responds except when required by law, so I recommend engaging regulators (FTC, and DOJ at the bare minimum) and legislators (primarily those focused on Section 230 reforms) whenever possible with regards to this entity. They're the only folks worth escalating to, as Reddit's incentives are to gate content, keep ad buyers happy, and keep the user base in check while they struggle to break even, sharing as little information publicly as possible along the way [1] [2].
[1] https://www.bloomberg.com/news/articles/2024-05-09/reddit-la... | https://archive.today/wQuKM
[2] https://www.sec.gov/edgar/browse/?CIK=1713445
No such thing, and definitely not the AP.
We thought it was an oversight too at first. It usually is. Large publishers have blocked us when they have not considered the details, but then reinstated us when we got in touch and explained.
Depends how you see it - if you see it as 'their' data (legally true) or if you see it as user content (how their users would likely see it).
If you see it as 'user content', they are actually selling the data to be abused by one company, rather than stopping it being abused at all.
From a commercial 'lets sell user data and make a profit' perspective I get it, although does seem short-sighted to decide to effectively de-list yourself from alternative search engines (guess they just got enough cash to make it worth their while).
Is that actually true? Reddit may indeed have a license to use that data (derived from their ToS), but I very much doubt they actually own the copyright to it. If I write a comment on Reddit, then copy-paste it somewhere else, can Reddit sue me for copyright infringement?
They demand that right. That doesn't mean they actually have a right to use the content in ways that are not directly required for the operation of the website or that are otherwise surprising to the average user. Putting something in the TOS doesn't always make it a valid contract.
you can always buy a competitor's or make your own vacuum cleaner if you hate buying at Walmart
maybe what you are really mad about is Reddit monopolising content
Similarly, if you start a vacuum cleaner company, you can make whatever exclusive deals you want. But if you control 80% of the market for vacuum cleaners, then you might need to be more careful about leveraging your market share in unfair ways.
If a company is part of a robust, competitive market (like Reddit), it's usually wiser to let customers vote with their wallets, and leave the government out of it. If a company becomes massively dominant (like Google or TicketMaster), and if it starts pushing exclusive contracts, it's much harder for customers to switch away.
You don’t even need >90% market share for that to be the case. e.g. Standard Oil only controlled 64% of the US market at most, it was still broken upz
Obviously, people do see value in it, or they wouldn't keep saying so! I would happily exclude Reddit links from search results though.
I don't consider the discussions there to be "real" in any meaningful way, thanks to the extensive moderation.
From what I've seen, there typically ends up being a small handful of moderator-enforced narratives that are deemed "acceptable" for a given subreddit, and any commenters deviating from those narratives get banned, or their comments end up as "[removed]" by "[deleted]", or the comments get obscured with the "comment score below threshold" notice.
It's generally some of the most one-sided and blandest discussion around. Given that there's often no meaningful back-and-forth involving differing perspectives of any sort, I'm not even sure if it should be considered "discussion". It's more like regurgitation and repetition.
I've found the situation to be particularly bad on the Canadian locale-specific subreddits, for example, but a enough of the tech-oriented ones I've seen seem to end up like that, too.
Not exactly always a reliable source of info outside uncontroversial niche topics or places like /r/AskHistorians that actually moderate. And even there I've seen the occasional humdinger.
※ - https://kagi.com/stats?stat=leaderboard&k=-2
Kagi shill here. Are they finally applying filters and operands to image searches?
Asking because it was a tough year seeing Pinterest as top filter choice and top result in images (when set as filter=block).
(edit: I just tried searching->image: beautiful quilt patterns. I didn't spot any Pinterest results!)
I have never understood why DDG, etc steadfastly refuse to obey operands in image searches. Most days. Every blue moon operands seem to work. I think.
sidebar: Yesterday I saw Yandex obey quotes in a web search. It was the 1st time I've seen that.
That was a bug, apologies. It should be fixed now.
Overall, my experience is very positive. I'm on many PCs throughout the day and I miss Kagi when it isn't there.
> If you need advanced blocking features, such as specifying rules by match patterns (e.g., ://.example.com/*) or by regular expressions (e.g., /example\.(net|org)/), the uBlacklist browser extension includes support for Kagi Search.
https://help.kagi.com/kagi/features/website-info-personalize...
This has greatly enhanced my Google experience - easy to ban content farms, AI-generator websites from appearing in Google Images etc.
Honestly, that makes those other engines way less valuable because for many topics, telling the engine to specifically narrow the results down to reddit comments is the only way to get a decent answer to what you're looking for. I'd definitely support blocking Quora from everything though.
https://techcrunch.com/2022/04/18/web-scraping-legal-court/
Eventually it becomes expensive to scrape reddit's data and most people will stop.
Many web scraping companies have loads of phones hooked up in a rack in order to use mobile IPs. Companies can't just block mobile IPs because their site would become unusable for several city blocks (mobile IPs often correspond to a specific cell tower). This is the face of modern web scraping: https://i.imgur.com/U2RXi5G.jpeg
Reddit's justification for this is profoundly wrong. Their "public content policy" is absurd doublespeak, and counter to everything the open internet is and hopes to be. You cannot simultaneously call yourself "open" and "public" while refusing access to automated clients. Every client is automated. They even go so far as to say that "crawling" (also known as "downloading") is an "abuse" and violates user privacy.
This is absurd, and not justified. I would love to see legislation that restricted server operators' ability to prohibit automated access in this way, but I suppose it will never happen. Some people in this thread have attempted to justify the policy by saying "they have to protect their income streams". No they don't. You don't have a right to an income stream, and you certainly don't have a right to lie in order to get all the benefits of an open internet with none of the downsides. Noting of course that the "downsides" are in this case actually just "competitors".
Or are you somehow suggesting that it’s google’s fault that Reddit took this step? I don’t see any indication that’s the case.
google is using their power to prevent others from competing.
the problem here is of course that if reddit would be in financial trouble (i don't know if they are but let's imagine they need this money), they'd be between a rock and a hard place.
google should not be allowed to make exclusive deals, and reddit could not survive without the deal, then what would be left? google buys reddit, or the relevant authority approves of the deal?
i thought about the same problem with firefox. let's assume firefox is forced to allow people to make a choice of the default search engine (just like microsoft was forced to allow a choice of default browser on windows) then google might stop paying mozilla, and they could end up in financial trouble.
ideally no company ever depends on a single other company that much, but that only works if we don't allow companies to grow that much in the first place.
let assume apple is forced to allow people to make a choice of the default search engine in safari then google might stop paying apple, and ...
Firefox has always allowed people to make a choice of the default search engine, since before it was even called Firefox. I know. I was there building it.
if the same was done for search engine choice for firefox then google would no longer be the default, and they would have no reason to pay firefox for that.
And as for Firefox, Mozilla being forced off Google's teat would be the best thing that could happen to the browser.
Google hasn't been a search engine in a long while, it's just an advertisement engine now.
it started when youtube removed the ability to search for videos older than 5 years, if I had to guess? cost saving, have every old video in cheaper storage... but it sort of fragments youtube, every couple of years you only get newer content.
But yeah, the most outrageous one is older videos, I do believe the reason is that they are using some long term cloud storage that is cheaper for older videos so they removed the ability to search by date.
Additionally, I don't believe the API fully fixes it, because Bing has a wrapper for youtube and searches do not really vary