It doesn't have to be quick for bad effects. See https://discussions.apple.com/thread/7908738 for example. I can't find the link now, but there was also an article about one of the older designers (IBM?) making sure the terminal cursor blinks in 1:2 ratio to reduce the problem.
Just don't give random unsuspecting people blinking images as a rule.
Same advice I gave a w3c.org admin who was lamenting how much traffic people generate by not caching xml schemas. Yes, you have to serve the requests. But you don't have to try to serve them in 100 ms. If a human is on the other end, 1-2 seconds is just fine. If a human is not, then the human will surely notice when their batch process goes from 3 minutes to 10 minutes because it fetches the same schema 200 times.
Well any time you start yanking levers and spinning dials you'd better know where the breaking points in your system are.
If you care about the traffic because you're already having trouble with that many simultaneous requests, then you are definitely not going to solve that problem by increasing the response time by a factor of 10.
But an important property of reverse proxies is that once the proxy sees the last byte of the response, the originating server is no longer involved in the transaction. The proxy server is stuck ferrying bits over a slow connection, and hopefully is designed for that sort of work load. If the payload is a static file, as it is in both of these cases, then it should be cheap for the server to retrieve them.
Yes, but slowloris isn't really a big deal if you've got a modern http(s) server with async i/o. It costs nearly nothing to have a idle connection while waiting 3 seconds before sendfiling the schema xml.
You can run out of sockets, but that's easy to tune. I don't know the limits on other systems, but FeeeBSD lets you set the maximum up to Physical Pages / 4 with just boot time setings. So about 1 million sockets per 16 GB of ram.
Worst case, if you start running out of sockets because you're sleeping, sample the socket count once a second and adjust sleep time to avoid hitting the cap. Also, you could use that sampling to drive decisions about keeping http sockets open or closed.
I should add, select on millions of sockets is going to suck; so you'll need kqueue/epoll/whatever your kernel select but better interface is.
It can't be a training pipeline, because the IPs are all around India.
Sample code from Stack Overflow being used by some major app is the most likely candidate. It's also possible that the image fetch call is a vestigial appendix that doesn't even display the image, which will make tracking this down extra challenging.
Wikimedia is unique in running some of the most popular websites with open access to almost all systems. As someone who has never been on the inside of FAANG, I found it rather interesting to browse around the backend infrastructure.
Wikimedia's infrastructure is radically different than most FAANG.
In large part because 99% (+/-) of their traffic is read only. While Facebook and Google have to do heavy workloads for every click and action taken on their services, Wikimedia can cache basically everything. Allowing them to operate on a tiny fraction of the number of machines (and infrastructure) that the rest of the players do.
You can do quite a bit of processing per page load without issue. Facebook and Google just take it rather past that point into near absurdity, while still being highly profitable.
I don't think that Facebook/Google developers are foolish or incompetent. That would be contempt. Instead, I think that Facebook and Google as conglomerate entities are fundamentally opposed to my right to privacy. That they make decisions to rationally follow their self-interest does not excuse the absurd lengths to which they go to stalk the general population's activities.
> I don't think that Facebook/Google developers are foolish or incompetent.
Nobody in this thread is saying that. Parent to you said:
> they could just stop being absurd instead [of building more DCs]
implying FB could build fewer DCs by scaling down some of their per-page complexity/"absurdity". Basically saying their needs are artificial or borne of requirements that aren't.
> conglomerate entities are fundamentally opposed to my right to privacy
That's a common view, but it's not on topic to this thread. This thread is mostly about the tech itself and how WikiMedia scales versus how the bigger techs scale. It has an interesting diversion into some of the reasons why their scaling needs are different.
You could instead continue the thread stating that they could save a lot of money and complexity while also tearing down some of their reputation for being slow and privacy-hostile by removing some of the very features these DCs support (perhaps) without ruining the net bottom line.
This continues the thread and allows the conversation to continue to what the ROI actually is on the sort of complexity that benefits the company but not the user.
I was the one saying absurdity and I think you’re missing the context. Work out how much processing power is worth even just another 1 cent per thousand page loads and perfectly rational behavior starts to look crazy to the little guys.
Let’s suppose the Facebook cluster spends the equivalent of 1 full second of 1 full CPU core per request. That’s a lot of processing power and for most small scale architectures likely adding wildly unacceptable latency per page load. Further, as small scale traffic is very spiky even low traffic sites would be expensive to host making it a ludicrous amount of processing power.
However, Google has enough traffic to smooth things out, it’s splitting that across multiple of computers and much of it is after the request so latency isn’t an issue, and it isn’t paying retail so processing power is little more than just hardware costs and electricity. Estimate the rough order of magnitude their paying for 1 second of 1 core per request and it’s cheap enough to be a rounding error.
The idea of marginal value/marginal cost is that companies will generally continue spending one billion dollars to add size and complexity, as long as they get back a bit more than a billion dollars in revenue.
So it wouldn't necessarily be contradictory if most of their core functionality could be replicated very simply, yet the actual product is immensely complicated. I forget where I first read this point, but probably on HN.
To be fair, there's a bit of a combinatoric effect of scale * features going on there. I'm sure you could build most of a Facebook equiv. 100x-1000x cheaper if it only served one city instead of the whole planet.
The effects of scale are less combinatoric than you might think. Most people on my Facebook feed are from the same city anyway, even though Facebook is global.
They also have looser latency SLAs. The only hard requirement is that a user can read back their own writes, but it’s okay if other users are served stale data for a few seconds or minutes even. This makes cache invalidation, one of the most notoriously difficult and expensive operations at large scale, much much easier.
Facebook also has a similar SLA. I've heard that at one point in their architecture (~2010), they literally stored the user's own writes in memcached and then merged them back into the page when rendered. You would see a page consistent with your actions, but if you logged into Facebook as any of your friends your updates might not show up until replication lag passed.
Close, IIRC we cached the fact you had just done a write, and a subsequent read request that arrived on the replica region was then proxied to the primary region instead of serviced locally.
My memory is fuzzy now but this dates back to when there were only two datacenter regions and one of them held all the primary DBs (2011 or so). All write endpoints were served in that region, so if a user routed to the secondary region did a write the request was proxied to the primary region. After doing a write a cookie was set for the user in question which caused any future reads to be proxied to the primary region for a few seconds while the DB replication stream (upon which cache invalidation was piggybacked) caught up, because if they went to the secondary region memcached was now stale.
It hasn’t been this way since around 2013 but again I am fuzzy on how. I think that’s when most such data was switched to TAO, which has local read what you wrote consistency. As long as users landed in the same cluster (and thus TAO cluster) what they wrote was visible to them, even if the DB write hadn’t yet replicated to their region.
FlightTracker postdates my time at FB (ended 2018ish) so I’m not sure how that is used. These systems evolved a lot over time as requirements changed.
I don’t remember anything about writes being batched in memcached and merged in on page load.
Search is also largely read-only. The advantage Wikipedia has is that its traffic overwhelming goes to the head of the page distribution, so simple caching solutions work very well. Google has a pretty extreme long-tail distribution (~15% of daily queries have never been seen before), and so needs to do a lot of computation per query.
> Google has a pretty extreme long-tail distribution (~15% of daily queries have never been seen before)
Do you have a source for this?
I'd be willing to bet that the ONLY reason why 15% of their daily queries "haven't been seen before" is because they add un-needed complexity like fingerprinting. You're making it seem like they've never seen a query for "cute animals" before when obviously they have. They choose to do a lot of extra leg work because of who you are.
So your claim that 15% of their queries have "never been seen before" is probably inaccurate. I'd be willing to bet that "15% of their queries are unique because of the user, location, or other external factor separate from the query itself."
They've seen your query before. They've just never seen you make this query from this device on this side of town before.
I think you mean "15% seems too high". Any easy way to think about this is the following: even if search the entire internet you will almost never see the same sentence twice, assuming it's has a certain number of words. There is a combinatorial explosion in possible sentences to write. Search queries are essentially just sentences without stopwords.
The point is not that. It’s that when you search for “cute animals”, Google shouldn’t be storing that you searched for that, or even care. Your location is arguably potentially relevant but it could be coarse enough except when searching for directions to allow at least some caching.
This is right on the money — getting search results for queries that are too personalized to e.g. location means that you can't cache those search results (or if you did cache them, their entries would be useless).
Hey Igor! Hate to be a bore, but I wanted to provide feedback that your comment may unintentionally come across as aggressive. OP has pretty relevant work experience that I know I’d love to hear more about, but there’s not really any room for them to respond.
I know many folks IRL who work at big tech who have no interest in posting here because the community comes off as very unwelcoming. That’s a shame, because they have insight that would be great to hear. Regardless of anyone’s opinion of their employer.
Apologies in advance if your intent was purely about the topic. I just thought I read something in your tone that might hinder discourse rather than encourage it. I wanted to point it out, in case it was unintentional.
Agreed about the tone. The comment could have been less argumentative — instead of "that's not the point," they could have said "that's not the only reason."
On the other hand, if I'm not responding, it's not because I find HN too abrasive — it's because I am afraid of leaking non-public information. That's why whenever I talk about Google, I try to cite a Google blog post or other authoritative source, or talk about my own personal experience; hence, "I rarely search for the same query twice."
I’m gonna have to disagree with the negative comments above concerning Igors tone. He made his point with clear respectful language that I would be happy to entertain at work, at the bar, at worship or while on a (previous to covid) group run or golf outing. so, to me, it looks like instead of an ‘agree to disagree’ while respecting each other, you disrespect igor by dismissing his arguments due to his tone, which handily allows you to ignore his content, such as it is. Therefore, in my judgement, you guys are being unfair to Igor while also being disingenuous about your reason for policing his tone.
"That's not the point" can be interpreted as respectful, but it also can be interpreted as argumentative. I chose to assume good intentions, but I offered a different phrasing that would have a higher chance of not being misinterpreted: i.e. using "yes and" instead of "no but": https://www.theheretic.org/2017/yes-and-vs-no-but/
I understand. To be a googler you have to be really good at smooth talking.
And when you do this on a public form it just highlights that the company you work for makes sure as many employees as possible are serving kool-aid to the masses.
I apologize for the tone. It the start of my comment was clumsily wired and it wasn’t my intention to have it come off as argumentative. The way I read the GP comment to mine was talking about how Google’s tracking of its users’ telemetry was what was contributing to the uniqueness of requests. Your comment to me boiled down to the fact that of course most requests are unique because of tracking location data and the user account. There seemed to be a disconnect because your comment took for granted that user location and account were a part of the search query while the person you were replying to specifically challenged that notion (again in my reading of both). I tried to post a concise bridge between the two concepts, and of course we all see how well I did with that :)
Having said that, I do think this is clearly a sensitive issues, not a purely technical one. I can appreciate the nuance of working for Google and doing excellent work while seeing the company criticized left and right for its business model. I think given the community, while there is opposition to how Google may at certain points conduct itself as a corporation, there is no lack of respect for any individual working there. I certainly view my comment and the discussion of privacy as having 50% to do with Google’s strategy and 50% to do with the technical aspects of whether you can build a search engine that holds user privacy as a core priority rather than trying to launch an ad hominem on you or anyone. And I saw your other comment that agreed with me and the GP comment so I think my first sentence aside, we are on the same page :)
To me Igors comment is also displaced. He injects activism into a technical discussion (sadly happens very often here on HN). We all know by now that the bigcorps are to a large degree based on data collection. We do not need to be reminded about it each and every day. We are adults, if we don't like it we use alternatives.
Yeah, this is a fair point. My larger point was mostly that HN misses out on some valuable comments by insiders because those people are disincentives by some of the rhetoric and tone when an article on big tech is popular. I didn’t think the comment I replied to was particularly aggressive - it was just something that came to mind when I read it. OP was actually very kind and constructive in their response - a good ending and constructive discussion for us all!
I agree with you. Also in my experience less tech-savvy people tend to overcomplicate their queries instead of just entering the relevant keywords which I'm sure accounts for many uniques.
I don't know how they did it but I suspect that it wouldn't be very hard to model the distribution by sampling a few million queries and extrapolate from that.
> Does it mean literally the text string typed into the box by the user is new?
I guess that could be the case. Many could be related to things that are on the news. Like, 'the cw powerpuff girls' for the new show that was announced. No one was searching for that until the announcement, probably
At the lower bound, that's 150 Billion "new" searches per year. There are approximately 50,000 unique english words not including names or misspellings. If google searches were on average for 3 words, it would take 833 years at that rate to go through all the combinations.
Alternatively, if we assume that google has already recorded 20 Trillion unique search queries (~ 1 Trillion new ones per year for 20 years), the odds that a query composed of 3 correctly spelled english words that are not names has been seen before is 1.6%. Even if we restrict queries to those using the most common 1000 words, there's a 50/50 chance of a query composed of 4 words being unique.
Of course people do not just type random words into the search bar and some terms will be searched many thousands or even millions of times, but still if anything the fact that 85% of searches aren't unique seems surprising.
I think GP post has a point. I've noticed people use Google really differently from how I do. E.g. I would go search for "figure concave" while my brother would search a longer phrase.
Also, speaking of people behaviour, it would not make sense to search everyday for "cute animals", but the volume of searches done for new things people discover as they get older would make more sense. I mean just look at search trends for things like "hydroxychloroquine" for example (and that's not to mention people who get it wrong, i.e. other factors for differing search queries too)
Also, other languages can change the queries depending on how you phrase the sentence too. Add to that the people using other ways to search instead of just visiting google.com and I think you can get pretty close to 10%.
If fingerprinting is the reason, 15% would be a figure too low I surmise. Would that be the case I think that would make probably 20-25% of searches rather than 15%.
It could very well be that they do classify fingerprinted search differently only in some countries and not others? That would/might explain the 15% figure.
I might be wrong and under-estimated fingerprinting techniques for Google. If they have really good fingerprinting techniques, that would reduce the estimate I have in mind to a better number (close to 15, maybe?)
So consider your hydroxychloroquine example again this way;
Nobody has ever searched for hydroxychloroquine before today. Today is the day the word is hypothetically invented. Today 2 million people will search for hydroxychloroquine. But only one of them was the first to do it.
What I know about pop-culture and viral internet culture is telling me that 15% of 1 trillion searches being unique is shady math.
So I am not fully convinced that the 15% claim is completely transparent.
It's a guess, but my thinking is that previously most people who searched term hydroxychloroquine were mainly scientists and other people related to that not your general population. Suddenly covid happens and now large numbers of people learn about this new drug they never heard before, they are gonna search, and I presume this, most wildly different things like:
"how does it work?"
"does it cause some disease?"
"insert something political here about hydroxychloroquine"
"did aliens make hydroxychloroquine?"
and many more things I lack imagination to come up with and that's only about hydroxychloroquine. I doubt 15% number is about single word cases, but more about combination of words and that seems reasonable. Inventing new words daily seems unlikely, chaining them on the other hand seems plausible.
The vast majority of people don't search for [hydroxychloroquine]. They search for [Is hydroxychloroquine effective in treating COVID-19?] or [What is the first drug that was approved to treat COVID-19?] or [What methods do we currently have to treat COVID-19?]. You can see these on the search results page as the "Common questions related to..." widget. How else do you think Google gets that data?
The folks who use keyword-based searches are largely those who got on the Internet before ~2007. Tech-savvy, relatively well-off, usually Millenial or Gen-X, plugged into trends. This happens to be the demographic dominant at Hacker News. But there's a much larger demographic who just types in whatever they're thinking of, in natural language, and expects to get answers.
Come to think of it, this is also the demographic that doesn't use tabbed browsing, and uses whichever browser ships with their OEM, and often doesn't realize that there's a separate program called a "browser" running when they click on the "Internet", and issues a Google Search for [google] (#3 query in 2010) when they want to get to Google even though they're on Google already but don't realize it, and doesn't know what a URL is. When a big-tech company makes a brain-dead usability decision you don't like, first consider how that usability choice might appear to your grandmother and it might not seem so brain-dead.
> So your claim that 15% of their queries have "never been seen before" is probably inaccurate.
I'm not sure, on my productive days maybe >50% of my Google searches are not very cachable. (for example, I just googled "htop namespace", "htop novel bytes", "htop pss", "htop nightly build ubuntu 14.04")
Wikimedia also has less incentive/drive to meticulously track every interaction on their pages. The level of tracking present on Facebook and Google has to be extremely computationally intensive.
I agree. Another (no contradiction) way of looking at this is that Wikimedia infrastructure is radically different because Wikimedia is radically different.
They need it to be a certain way in order to operate. The limitations and advantages of how software gets made. Why it gets made. The way the software works. How and why product decisions were made over the last 2 decades. What resources they have/had available. It's all a totally different game. Not surprising that different soil and a different climate grow different plants.
One of Google's early coup d'etats, when they were a strategic step ahead of the boomers, was bankrolling gmail, youtube and such. Gmail offered free giant inboxes. They got all the customers. This cost billions (maybe 100s of millions), but storage costs go down every year while the value of ads/data/lock-in and such go up every year. Similar logic for youtube. (1) Buy a leading video-sharing site; (2)bankroll HD streaming because you have the deepest pockets (3) Own online free TV entirely.
That's who Google is, good or bad. How funding works. What products get built. What infrastructure is necessary, possible, affordable. All interlinked. Wikipedia & Google were founded at the same time. Within 5 years (circa 2006) Google was buying charters and fiefdoms. Wikimedia, meanwhile, was starting to take flak for raising 3 or 4 million in donations.
It's kinda crazy that Wikipedia is comparable in scale to FAANGs when you consider these disparities.
Looking at the source of a Wikipedia page it has my username appearing 6 times so I guess it must reduce caching a bit. Though I guess they could cache the user info bits and the rest of the page and just splice them together.
In interviews with Jimmy Wales, he seems somewhat regretful of not having made Wikipedia a for-profit. At the least, he's fairly adamant that Wikipedia could have been Wikipedia as a for profit.
The way he structured wikipedia, from back-end infrastructure to ownership/governance structure was just the logical way of doing the project. Times were different. Online culture was different.
I don't want to overinterpret the man, or put words in his mouth... but... I got the impression that Wales thinks that if he was starting Wikipedia now, he'd just do it asd a startup and also succeed.
To me, this is almost sad. Besides being an awesome encyclopedia, wikipedia is existence proof for something of scale outside the norm. Something that isn't a corporation. A lot of things are deterministic to the structure of an organization.
For example, take the current postpostmodern war over truth and stuff: platforming/deplatforming, freedom of speech, censorship, bias, manipulation, narrative = power issues, etc. Wikipedia is at the very centre all these problems. Whatever difficulties Twitter is experiencing should be 100X worse for wikipedia. Meanwhile, Wikipedia is withstanding far better, and with far more integrity. I don't think this is a coincidence.
Dunking on wikipedia's budget/spending is popular. Meanwhile, Wikipedia uses <1% of the resources/budget of Twitter. They are operating @ >100X efficiency compared to a realistic for-profit equivalent. That's a flying shuttle.
We know that Wikipedia, Linux & The Worldwide Web are possible because they exist. We literally wouldn't know otherwise. Theory couldn't have gotten us to this knowledge. Each is existence proof for other ways of doing things. They aren't necessarily roadmaps, but I'm a big believer in existence proofs. What Jimmy made is 100X better, more important and non-inevtiable than what Zuck made. The thought that he wants to be Zuck bums me out.
It would succeed the same way Quora does. Much less open, much less universal, much more user hostile, with an almost agressive way to deal with unlogged user.
In terms of financial and organisational success it would probably largely beat what it is now. It terms of benefit to humanity, it would be much worse.
Company + for profit + laws means access to information has to be much more tailored to the laws of each place. "Let's remove tianamen's article or lose your chinese license" kind of things.
I'm for one am glad for the current wikipedia we have, despite it's numerous flaws. I still donate every year, although I wish Wales could stop having it spend its money the same a startup or FAANG does.
That's one option, though I wouldn't necessarily use Quora as a mainline example. They're kind of a $gme for rich people. I think highly enough of Jimmy to bet on him doing a much better job than that.
Stackoverflow is a decent example. Very capable founding team. They explicitly tried to be like a commercial wikimedia. They do embrace quite a lot of openness, notably creative commons... learning from wikimedia successes.
RE "I wish Wales would:" Another consequence for how wikipedia is structured is that Wales isn't the Zuckerberg of Wikimedia. Power is a lot more dispersed.
RE spending/flaws and such: I feel like wikimedia is held to an extremely unfair standard. Who/what should we compare them to?
Wikimedia spend $70m per year. This is probably less than Quora or stackexchange. FB & Twitter (IMO more comparable in terms of scale/importance) spend $55bn & $3bn. Twitter spends 45X more than Wikimedia. Facebook spends almost 1,000X compared to Wikimedia. The bang-for-buck is insane.
Also in terms of flaws in rules/judgement calls. A lot of people are highly critical of wikipedia's "deletionism" related MOs. What articles/edits stay in. How good the rules & procedures are for this. What "camp" has power, and how they treat the other camp. I get that this stuff is contentious.
Meanwhile on Twitter or Facebook, the rule is "I decide." "But it gets us clicks" is the killer argument. Nothing is transparent. Wikimedia is doing a much better job, respecting user & editor rights far more, being a lot less self righteous. Of course it's not perfect, but come on. The "norm" is Facebook's content policy, Twitter's safety department, or Apple's App store approval room. Wikimedia is the one example of being better than that... and for that everyone is always yelling at them.
Quora imo is a horrible website and I rarely find actually good advice on it. At this point I actively avoid clicking on it's links because of how aggressive they are towards non logged users.
I can also imagine that he'd say that just because it makes him look/feel better, i.e. it's more of a sacrifice if he gave it for free while he also could've been a billionaire, than if this was the only way Wikipedia could ever have been a success.
Yeah, the Web was quite impressive (though we already had the Minitel), but it was Wikipedia that really blew my mind (even though we already had Encarta). (In fact I consider Wikipedia to be the Web's "killer app", even more than Google and other search engines were.)
Out of all the "killer apps" for the web... wikipedia is the one that implement the www most faithfully. Hypertext articles. Most apps got the web to do x. Wikipedia is what it was made to do.
Yeah. I was about to add that it pretty much has been Tim Berners-Lees vision coming to fruition, but the fact that Wikipedia is centralized has stopped me. But then isn't the Web itself technically 'centralized' on the Internet ? And isn't Wikipedia a great example of pseudonymous strangers (= social decentralization) collaborating with each other ?
Is it really that centralized? Citations and footnote links are a pretty important part of what wikipedia is. I mean, I very rarely click through to see source material, but when I do it's noticeable how much more powerful wikipedia is than a standard encyclopedia.
At one point, I had a very small png file that was large enough to crash Netscape and IE, and later firefox. It had large enough dimensions that browsers couldn't handle it.
Today I'm sure it would be fine; instead I'm frustrated by my inability to create webp images larger than 16000 pixels tall (i was trying to write a data-saver proxy for reading webtoons)
With the amount of traffic its apparently getting, HN probably won't make a big impact. Plus, most of us aren't in India, and most of us have normal user agents.
That’s a good one! Maybe an app which generates this you’r type of image has this flower as one of their sample images in a list which it preload on startup
A permanent redirect to a non-image page (owned by Wikimedia) may achieve the same thing. Either the calling system can't support a HTML response, or it's a webview in which case you could either report an error or provide a notice. Maybe even ask for donations :)
Yes there is when you are hotlinking. Hotlinking in general is considered theft, you are using someone elses bandwidth and could even ddos the host if you are not caching the response.
it's so easy to mitigate, though, that the fact that one doesn't sorta implies that one might want randos from the internet to use one's resources to view this image.
it's not theft if you leave it out for everyone to use.
No, but if I wander into your garden and "injure" myself, I can sue you for damages. You will be held negligent for not properly protecting yourself from preventing other people from injuring themself on your property.
The problem of course is that the "victim" has a lawyer operating on a contingency, whereas you have to pay your legal costs, and generally cannot recuperate them.
In France (at least), all swimming pools are protected by a fence. If you own a pool and don't put a fence around it, you can be held responsible for a child drowning into it.
It is possible this principle applies to other countries and other things than pools.
Here in Russia, if you leave poisonous chemicals like methanol, etc, unmarked or put a bear trap in your locked house behind a locked fence with a generic warning sign, and then someone dies or gets injured by these, chances are you will go to jail. Idk if this applies to accidental traps like pools or rakes in grass. Same for taking a knife out of an attackers hand and stabbing them back. (Yes, our laws protect criminals better than citizens, not joking.)
Interesting. So if I understand this correctly, if someone breaks into your house and gets injured, and they can make a good case for some kind of negligence on your part, then they can successfully sue you?
It's also illegal to set a trap in your own home in the US as well, decided when a property owner, tired of people breaking into his property while he was away, set up a shotgun booby trap that injured a burglar.
https://youtu.be/bV9ppvY8Nx4
I wasn't sure if it is the same or similar principle in Russia or a different one that requires active care for a burglar. Unlabeled chemicals causing liability for a burglar seems extreme to me
Leaving a bear trap goes way beyond negligence, it's literally setting a trap. Similar with unmarked dangerous chemicals, they're required to be marked for good reason.
In Poland setting marked traps on your own, fenced property is illegal and their owner is responsible for any harm they cause, because there exist legal reasons to enter another person's property - for example to fight spreading fire.
However my favourite example is the law that allows any bee keeper to enter any private property if they are pursuing fleeing bee swarm.
Wikimedia has a User-Agent policy which is being violated here. Hence this is the property owner putting up a sign that says "risk of injury", so if you walk in and injure yourself, you only have to blame yourself for being negligent.
It's a policy how wikimedia acts when clients lack a user agent header, it's therefore effectively a rule for clients as without a proper UA header, they may be blocked indefinitely.
You think this, but how much experience do you have with it? People know that homeowners have insurance. They sue to make the insurance pay out. It happened to my neighbor. So you can make all of the dumb countries comments you want, but it doesn't make it any less real.
This is a pretty puzzling idea to me. How could linking something be theft?
To explore this, I shall try a metaphor. Imagine you're on a big social media website (lets call it Programmer Olds) which has an oddity in that 99% of its users use adblock. You then post a link to another small (ad supported) website on your Programmer Olds page, causing a large number of people to click through and download the page using large amounts of bandwidth (for no monetary gain to the site) and possible DDOSing the site.
> causing a large number of people to click through and download the page using large amounts of bandwidth (for no monetary gain to the site)
The difference here is that while a lot of users use adblock, there are some that don't. These users can still see the ads. Additionally even though it's a small website, it may lead to new readers that stick around or the content itself may even be sponsored.
The equivilent to hot linking a picture would be like taking the content of a blog post without really linking to the source, because there's no chance of conversions there. If you're linking to the site itself then there's a reasonable chance that users can convert.
So I suggest that it's theft just because the chances of readers being converted is nil while you're using their bandwidth.
That's because you're responding to an entirely different issue. "Hotlinking" isn't linking to something, it's including a resource that is hosted elsewhere. It's putting <img src="https://concordDance.whatever/images/big_image.jpg"> on my website without asking you. Now if my site ends up on the front page of HN, that could cause a lot of traffic to your site, potentially overwhelming your server or increasing your hosting bill. It's not nice, and rightfully frowned upon.
Let's say I own a restaurant. Someone comes in and wants a panini. I don't have a panini press, but the restaurant next door does.
If I tell the customer they can go next door to get a panini, I'm not stealing anything. Maybe that restaurant is packed right now and they'ed rather not have an extra customer, but there is a reasonable expectation that they would generally want customers or at least have a means of turning away unwanted customers otherwise.
On the other hand if I break into my neighbor's restaurant, make a panini, then bring it back to my restaurant to serve and make money off of, all without permission from the neighbor, I am most definitely stealing. Even if I doubt the neighbor will mind because he let me come over and make myself a panini once, I can't unilaterally act off that assumption.
...or sending that image[1] jwz sends back upon detecting HN in the referer. I bet they'll find the app in a matter of hours, or at least reduce the traffic drastically.
This makes me wonder why the hell referer headers are still sent by major browsers, especially to third parties. I can’t think of a single reason that benefits the user.
Originally it probably just sounded like a cool feature to see what blog linked to you. Now its been around for so long that so much has been programmed to actually use it. If you turn it off you get every anti bot script blowing up on you.
I think browsers did drop the path from it at least.
For one thing, examining referer is a common way that a server determines a request is not a hotlink. Sure you can do something more complicated with cookies or whatever, but lots of sites are just using referer and they'll break if the client doesn't send it.
Well, it'd mean that any site blocking hotlinking would also automatically block direct bookmarks/URL entry, too, which isn't really in the "interest of the user" either, I'd say.
I saw the described image but after I visited the site directly I couldn't see it any more when redirectly via hacker news. Saw it again when I opened an incognito tab.
Just learned that this person owns DNA Lounge (and pizza?), and is a founder (early contributor?) of Netscape and Mozilla.org. I've lived and worked in that particular area of SF for years and haven't known this.
One of my company's clients has a beautiful office right above DNA Lounge (well, across the street or just adjacent - it's been a while and I've only been there once). They told me they can hear sound checks from their rooftop patio.
about:robots is from the early Firefox releases. Pretty sure it is from Firefox 3.0 development as you can find the same robot in images when searching for Firefox Gran Paridiso Robot.
It's a motivational-poster-type image with a white egg holder in the foreground, but instead of an egg, it's holding one exquisitely detailed hairy, caucasian ball[1]. At the top, the title is "HACKER NEWS" and the bottom text is "A DDoS OF FINANCE-OBSESSED MAN-CHILDREN AND BROGRAMMERS"
1. Is there a collective biological term for scrotum and it's contents that is not general like "genitals" is?
I missed the edit window and I’m disappointed in myself for mentioning it by name. Please just don’t Google this unless you’re prepared for an upsetting image, and even then maybe just skip it. You’re probably not as prepared as you think.
Big stretched open butthole. Not sure if you need the warning but I’m commenting in case anyone would prefer not to see it despite their curiosity.
Sorry to ruin the fun y’all but there’s images I won’t even mention that I can’t unsee and make me feel seriously ill when I do see them. I don’t want anyone else to feel that way without warning.
I remember when I was about 15, before pop-up blockers were really a thing, someone sent me a link to that and it would keep opening popups with that image and you couldn't close all of them :-/
Sometimes people look back to the internet of the 90s with too rose-coloured glasses IMO.
Hey at least if you were on a 90s Mac your computer was probably unresponsive and you could skip to the inevitable force reboot. And browsers didn’t save sessions so you were in the clear as soon as you got to tabula rasa.
I’m honestly not sure you’re asking in good faith so I’m not going to add more (and if you are asking in good faith you’ve got plenty in responses to go on). Also I never knew the name of the one that’s permanently burned into my brain and I’m so glad I don’t.
Why does it need to be fixed? The mission of wikimedia is to serve educational content.
Edit: this is a bit unfair, if its a specific app they should be convinced to cache just to avoid unfair resource usage, but hotlinking in general should not be seen as a problem
Any for-profit entity hotlinking Commons is unfair. Heck, they have the right to redistribute freely the image as they see fit, instead of consuming resources that are a common good.
But this goes beyond that - it's some blind check of internet connectivity for the app, and doesn't get shown to the user. We're pretty sure of that, given that with the amount of noise that task generated, if there was an app featuring that image at least one of the ~ 90M daily "views" would've been someone reading these posts.
I am a conservative man of tradition; fuck these modern liberal commies and their new age ways calling their degenerate diplomatic solutions “enlightenment”.
After realizing "wiki[p|m]edia" and "flower" triggered a specific image in my head I was guessing it would be a yellow flower, this one in the corner of https://www.mediawiki.org/wiki/MediaWiki but nope, more interesting than that!
Huh, I worked on a site with a similar issue in ~2019. A massive flood of traffic for a single site from Indian mobile apps (~15kqps at peak iirc).
I think it ended up being a sort of mobile-based botnet with a bizarre target, which luckily was deduced from some of the headers sent (they all had a random common header).
I was curious what the actual amount of requests HN would have to muster, and with a frequency of 90,000,000 reqs/day, HN would need to hit it with 4,500,000 requests.
I had some random images on a web server years ago - and noticed that something like 99% of my traffic was one image - and searching through refers I realized I was the #1 hit on google images for robot attack cat.
AFAIK both of these are using methods that are either deemed acceptable by adblockers (like how Google was using plain blue/black text instead of stroboscopic gif banners), and/or adblockers have trouble with them because they come from the same source ?
They have an app which is ad supported, and they are trying to become of a social network like reddit by requiring you to login for more features, and adding upvotes and other interactivity.
There's a site that's occasionally posted in comments here which is apparently run by someone who hates HN because they serve some image I can't remember to readers from here.
I had a similar issue. Some 15+ years ago, an image from my blog showed up for people who searched the phrase 'Peanubutter Sex'. The image had nothing to do with peanutbutter nor with sex. My blog is SFW. It was some screenshot of KDE IIRC.
For almost a week it remained the most requested image the post on which it appeared, the most popular.
It did make me uncomfortable, though. Fearing that my rankings would plummet or so.
My takeaway is nothing new: there are weirdo's online.
My blog did have the word "Peanutbutter" on one or two posts. And the word "sex" on another. Maybe at some point both words showed up close to one another when experimenting with some "random articles" sidebar or some "you may also like" list.
I'll just be looking forward to the follow-up post on HN announcing when they figure out what the culprit was!
Per the comments, right now the top suspect seems to be the app "Josh" or another TikTok clone because of how traffic surged immediately after the TikTon ban:
Nice idea, but if it's getting 90 million requests per day, then either there are a lot of people requesting the same message (so it's not very secret), or the few people requesting it are very forgetful (in that they have to keep re-requesting the same message multiple times per day).
I suppose the contents of the image/"message" could change every day, but presumably that would be very obvious in the edit history of that file[0], unless Wikimedia Commons were suppressing the fact that the file is constantly changing. If they are part of the conspiracy, though, you'd think they would have taken down the task from Phabricator too.
You gotta respect the suggested approach to take preventive measures by banning requests to this individual image without a User Agent header and to try to identify who might be affected. I’m sure I’m not the only one here who would just treat it as abuse and ban without followup.
386 comments
[ 3.0 ms ] story [ 316 ms ] threadOnly 10^-6 of the "legitimate" requests would be affected, but a whole lot of the "undesireable" requests would see it...
There was a better idea posted in comments - serve a picture with a very short explanation and an email to contact.
Just don't give random unsuspecting people blinking images as a rule.
Same advice I gave a w3c.org admin who was lamenting how much traffic people generate by not caching xml schemas. Yes, you have to serve the requests. But you don't have to try to serve them in 100 ms. If a human is on the other end, 1-2 seconds is just fine. If a human is not, then the human will surely notice when their batch process goes from 3 minutes to 10 minutes because it fetches the same schema 200 times.
I guess a couple seconds won't matter unless the server is already redlining it and the tarpitted traffic is a small proportion.
If you care about the traffic because you're already having trouble with that many simultaneous requests, then you are definitely not going to solve that problem by increasing the response time by a factor of 10.
But an important property of reverse proxies is that once the proxy sees the last byte of the response, the originating server is no longer involved in the transaction. The proxy server is stuck ferrying bits over a slow connection, and hopefully is designed for that sort of work load. If the payload is a static file, as it is in both of these cases, then it should be cheap for the server to retrieve them.
Worst case, if you start running out of sockets because you're sleeping, sample the socket count once a second and adjust sleep time to avoid hitting the cap. Also, you could use that sampling to drive decisions about keeping http sockets open or closed.
I should add, select on millions of sockets is going to suck; so you'll need kqueue/epoll/whatever your kernel select but better interface is.
Sample code from Stack Overflow being used by some major app is the most likely candidate. It's also possible that the image fetch call is a vestigial appendix that doesn't even display the image, which will make tracking this down extra challenging.
See, for example, their statistics at https://grafana.wikimedia.org/d/000000102/production-logging...
In large part because 99% (+/-) of their traffic is read only. While Facebook and Google have to do heavy workloads for every click and action taken on their services, Wikimedia can cache basically everything. Allowing them to operate on a tiny fraction of the number of machines (and infrastructure) that the rest of the players do.
The contempt on here is crazy sometimes.
Nobody in this thread is saying that. Parent to you said:
> they could just stop being absurd instead [of building more DCs]
implying FB could build fewer DCs by scaling down some of their per-page complexity/"absurdity". Basically saying their needs are artificial or borne of requirements that aren't.
> conglomerate entities are fundamentally opposed to my right to privacy
That's a common view, but it's not on topic to this thread. This thread is mostly about the tech itself and how WikiMedia scales versus how the bigger techs scale. It has an interesting diversion into some of the reasons why their scaling needs are different.
You could instead continue the thread stating that they could save a lot of money and complexity while also tearing down some of their reputation for being slow and privacy-hostile by removing some of the very features these DCs support (perhaps) without ruining the net bottom line.
This continues the thread and allows the conversation to continue to what the ROI actually is on the sort of complexity that benefits the company but not the user.
Let’s suppose the Facebook cluster spends the equivalent of 1 full second of 1 full CPU core per request. That’s a lot of processing power and for most small scale architectures likely adding wildly unacceptable latency per page load. Further, as small scale traffic is very spiky even low traffic sites would be expensive to host making it a ludicrous amount of processing power.
However, Google has enough traffic to smooth things out, it’s splitting that across multiple of computers and much of it is after the request so latency isn’t an issue, and it isn’t paying retail so processing power is little more than just hardware costs and electricity. Estimate the rough order of magnitude their paying for 1 second of 1 core per request and it’s cheap enough to be a rounding error.
So it wouldn't necessarily be contradictory if most of their core functionality could be replicated very simply, yet the actual product is immensely complicated. I forget where I first read this point, but probably on HN.
Edit: I don’t know what I’m talking about. Happy Monday!
Also, there's way more than just the web tier out there.
Everyone, please listen to Rachel and never ever me.
It hasn’t been this way since around 2013 but again I am fuzzy on how. I think that’s when most such data was switched to TAO, which has local read what you wrote consistency. As long as users landed in the same cluster (and thus TAO cluster) what they wrote was visible to them, even if the DB write hadn’t yet replicated to their region.
FlightTracker postdates my time at FB (ended 2018ish) so I’m not sure how that is used. These systems evolved a lot over time as requirements changed.
I don’t remember anything about writes being batched in memcached and merged in on page load.
Do you have a source for this?
I'd be willing to bet that the ONLY reason why 15% of their daily queries "haven't been seen before" is because they add un-needed complexity like fingerprinting. You're making it seem like they've never seen a query for "cute animals" before when obviously they have. They choose to do a lot of extra leg work because of who you are.
So your claim that 15% of their queries have "never been seen before" is probably inaccurate. I'd be willing to bet that "15% of their queries are unique because of the user, location, or other external factor separate from the query itself."
They've seen your query before. They've just never seen you make this query from this device on this side of town before.
15% of the queries themselves are unique. https://blog.google/products/search/our-latest-quality-impro...
https://www.google.com/search/howsearchworks/responses/
I work for Google (and used to work on Search).
Internet users who came online later, from GenZ to many boomers, will often just write conversational sentences and questions.
I know many folks IRL who work at big tech who have no interest in posting here because the community comes off as very unwelcoming. That’s a shame, because they have insight that would be great to hear. Regardless of anyone’s opinion of their employer.
Apologies in advance if your intent was purely about the topic. I just thought I read something in your tone that might hinder discourse rather than encourage it. I wanted to point it out, in case it was unintentional.
On the other hand, if I'm not responding, it's not because I find HN too abrasive — it's because I am afraid of leaking non-public information. That's why whenever I talk about Google, I try to cite a Google blog post or other authoritative source, or talk about my own personal experience; hence, "I rarely search for the same query twice."
I didn't dismiss his argument; I said that he was correct right after he posted: https://news.ycombinator.com/item?id=26073488
"That's not the point" can be interpreted as respectful, but it also can be interpreted as argumentative. I chose to assume good intentions, but I offered a different phrasing that would have a higher chance of not being misinterpreted: i.e. using "yes and" instead of "no but": https://www.theheretic.org/2017/yes-and-vs-no-but/
And when you do this on a public form it just highlights that the company you work for makes sure as many employees as possible are serving kool-aid to the masses.
Having said that, I do think this is clearly a sensitive issues, not a purely technical one. I can appreciate the nuance of working for Google and doing excellent work while seeing the company criticized left and right for its business model. I think given the community, while there is opposition to how Google may at certain points conduct itself as a corporation, there is no lack of respect for any individual working there. I certainly view my comment and the discussion of privacy as having 50% to do with Google’s strategy and 50% to do with the technical aspects of whether you can build a search engine that holds user privacy as a core priority rather than trying to launch an ad hominem on you or anyone. And I saw your other comment that agreed with me and the GP comment so I think my first sentence aside, we are on the same page :)
I doubt you store the history of all searches ever? People don't need a google account to query the engine, others disable history, etc.
Are you saying you still have all searches ever made ever? Because you would need this to say a query hasn't been made before wouldn't you?
"There are trillions of searches on Google every year. In fact, 15 percent of searches we see every day are new"
Does it mean literally the text string typed into the box by the user is new?
Or does it mean the text string combined with a bunch of other inferred parameters we don’t know about is new?
I guess that could be the case. Many could be related to things that are on the news. Like, 'the cw powerpuff girls' for the new show that was announced. No one was searching for that until the announcement, probably
Alternatively, if we assume that google has already recorded 20 Trillion unique search queries (~ 1 Trillion new ones per year for 20 years), the odds that a query composed of 3 correctly spelled english words that are not names has been seen before is 1.6%. Even if we restrict queries to those using the most common 1000 words, there's a 50/50 chance of a query composed of 4 words being unique.
Of course people do not just type random words into the search bar and some terms will be searched many thousands or even millions of times, but still if anything the fact that 85% of searches aren't unique seems surprising.
They briefly mention the statistic in the last paragraph.
Also, speaking of people behaviour, it would not make sense to search everyday for "cute animals", but the volume of searches done for new things people discover as they get older would make more sense. I mean just look at search trends for things like "hydroxychloroquine" for example (and that's not to mention people who get it wrong, i.e. other factors for differing search queries too)
Also, other languages can change the queries depending on how you phrase the sentence too. Add to that the people using other ways to search instead of just visiting google.com and I think you can get pretty close to 10%.
If fingerprinting is the reason, 15% would be a figure too low I surmise. Would that be the case I think that would make probably 20-25% of searches rather than 15%.
It could very well be that they do classify fingerprinted search differently only in some countries and not others? That would/might explain the 15% figure.
I might be wrong and under-estimated fingerprinting techniques for Google. If they have really good fingerprinting techniques, that would reduce the estimate I have in mind to a better number (close to 15, maybe?)
Nobody has ever searched for hydroxychloroquine before today. Today is the day the word is hypothetically invented. Today 2 million people will search for hydroxychloroquine. But only one of them was the first to do it.
What I know about pop-culture and viral internet culture is telling me that 15% of 1 trillion searches being unique is shady math.
So I am not fully convinced that the 15% claim is completely transparent.
The folks who use keyword-based searches are largely those who got on the Internet before ~2007. Tech-savvy, relatively well-off, usually Millenial or Gen-X, plugged into trends. This happens to be the demographic dominant at Hacker News. But there's a much larger demographic who just types in whatever they're thinking of, in natural language, and expects to get answers.
Come to think of it, this is also the demographic that doesn't use tabbed browsing, and uses whichever browser ships with their OEM, and often doesn't realize that there's a separate program called a "browser" running when they click on the "Internet", and issues a Google Search for [google] (#3 query in 2010) when they want to get to Google even though they're on Google already but don't realize it, and doesn't know what a URL is. When a big-tech company makes a brain-dead usability decision you don't like, first consider how that usability choice might appear to your grandmother and it might not seem so brain-dead.
I'm not sure, on my productive days maybe >50% of my Google searches are not very cachable. (for example, I just googled "htop namespace", "htop novel bytes", "htop pss", "htop nightly build ubuntu 14.04")
What can be interesting, I think, is that you have a completely open infrastructure that has to solve problems on a global traffic scale.
If people are interested in knowing more, I suggest you also take a peek at the wikimedia techblog, specifically to the SRE category https://techblog.wikimedia.org/category/site-reliability-eng... and the performance one https://techblog.wikimedia.org/category/performance/
They need it to be a certain way in order to operate. The limitations and advantages of how software gets made. Why it gets made. The way the software works. How and why product decisions were made over the last 2 decades. What resources they have/had available. It's all a totally different game. Not surprising that different soil and a different climate grow different plants.
One of Google's early coup d'etats, when they were a strategic step ahead of the boomers, was bankrolling gmail, youtube and such. Gmail offered free giant inboxes. They got all the customers. This cost billions (maybe 100s of millions), but storage costs go down every year while the value of ads/data/lock-in and such go up every year. Similar logic for youtube. (1) Buy a leading video-sharing site; (2)bankroll HD streaming because you have the deepest pockets (3) Own online free TV entirely.
That's who Google is, good or bad. How funding works. What products get built. What infrastructure is necessary, possible, affordable. All interlinked. Wikipedia & Google were founded at the same time. Within 5 years (circa 2006) Google was buying charters and fiefdoms. Wikimedia, meanwhile, was starting to take flak for raising 3 or 4 million in donations.
It's kinda crazy that Wikipedia is comparable in scale to FAANGs when you consider these disparities.
I recall reading HackerNews used to have that problem, unsure if it still does.
The way he structured wikipedia, from back-end infrastructure to ownership/governance structure was just the logical way of doing the project. Times were different. Online culture was different.
I don't want to overinterpret the man, or put words in his mouth... but... I got the impression that Wales thinks that if he was starting Wikipedia now, he'd just do it asd a startup and also succeed.
To me, this is almost sad. Besides being an awesome encyclopedia, wikipedia is existence proof for something of scale outside the norm. Something that isn't a corporation. A lot of things are deterministic to the structure of an organization.
For example, take the current postpostmodern war over truth and stuff: platforming/deplatforming, freedom of speech, censorship, bias, manipulation, narrative = power issues, etc. Wikipedia is at the very centre all these problems. Whatever difficulties Twitter is experiencing should be 100X worse for wikipedia. Meanwhile, Wikipedia is withstanding far better, and with far more integrity. I don't think this is a coincidence.
Dunking on wikipedia's budget/spending is popular. Meanwhile, Wikipedia uses <1% of the resources/budget of Twitter. They are operating @ >100X efficiency compared to a realistic for-profit equivalent. That's a flying shuttle.
We know that Wikipedia, Linux & The Worldwide Web are possible because they exist. We literally wouldn't know otherwise. Theory couldn't have gotten us to this knowledge. Each is existence proof for other ways of doing things. They aren't necessarily roadmaps, but I'm a big believer in existence proofs. What Jimmy made is 100X better, more important and non-inevtiable than what Zuck made. The thought that he wants to be Zuck bums me out.
In terms of financial and organisational success it would probably largely beat what it is now. It terms of benefit to humanity, it would be much worse.
Company + for profit + laws means access to information has to be much more tailored to the laws of each place. "Let's remove tianamen's article or lose your chinese license" kind of things.
I'm for one am glad for the current wikipedia we have, despite it's numerous flaws. I still donate every year, although I wish Wales could stop having it spend its money the same a startup or FAANG does.
Stackoverflow is a decent example. Very capable founding team. They explicitly tried to be like a commercial wikimedia. They do embrace quite a lot of openness, notably creative commons... learning from wikimedia successes.
RE "I wish Wales would:" Another consequence for how wikipedia is structured is that Wales isn't the Zuckerberg of Wikimedia. Power is a lot more dispersed.
RE spending/flaws and such: I feel like wikimedia is held to an extremely unfair standard. Who/what should we compare them to?
Wikimedia spend $70m per year. This is probably less than Quora or stackexchange. FB & Twitter (IMO more comparable in terms of scale/importance) spend $55bn & $3bn. Twitter spends 45X more than Wikimedia. Facebook spends almost 1,000X compared to Wikimedia. The bang-for-buck is insane.
Also in terms of flaws in rules/judgement calls. A lot of people are highly critical of wikipedia's "deletionism" related MOs. What articles/edits stay in. How good the rules & procedures are for this. What "camp" has power, and how they treat the other camp. I get that this stuff is contentious.
Meanwhile on Twitter or Facebook, the rule is "I decide." "But it gets us clicks" is the killer argument. Nothing is transparent. Wikimedia is doing a much better job, respecting user & editor rights far more, being a lot less self righteous. Of course it's not perfect, but come on. The "norm" is Facebook's content policy, Twitter's safety department, or Apple's App store approval room. Wikimedia is the one example of being better than that... and for that everyone is always yelling at them.
Then again, WikiTribune was a for-profit.
Today I'm sure it would be fine; instead I'm frustrated by my inability to create webp images larger than 16000 pixels tall (i was trying to write a data-saver proxy for reading webtoons)
https://www.wsj.com/articles/the-internet-is-filling-up-beca...
I'm sure there is a more enlightened fix.
it's not theft if you leave it out for everyone to use.
But you can totally sue anyone for anything, and that makes for entertaining headlines - even though if plaintiff lost promptly
It is possible this principle applies to other countries and other things than pools.
I wasn't sure if it is the same or similar principle in Russia or a different one that requires active care for a burglar. Unlabeled chemicals causing liability for a burglar seems extreme to me
However my favourite example is the law that allows any bee keeper to enter any private property if they are pursuing fleeing bee swarm.
This is a pretty puzzling idea to me. How could linking something be theft?
To explore this, I shall try a metaphor. Imagine you're on a big social media website (lets call it Programmer Olds) which has an oddity in that 99% of its users use adblock. You then post a link to another small (ad supported) website on your Programmer Olds page, causing a large number of people to click through and download the page using large amounts of bandwidth (for no monetary gain to the site) and possible DDOSing the site.
Have you commited theft?
The difference here is that while a lot of users use adblock, there are some that don't. These users can still see the ads. Additionally even though it's a small website, it may lead to new readers that stick around or the content itself may even be sponsored.
The equivilent to hot linking a picture would be like taking the content of a blog post without really linking to the source, because there's no chance of conversions there. If you're linking to the site itself then there's a reasonable chance that users can convert.
So I suggest that it's theft just because the chances of readers being converted is nil while you're using their bandwidth.
That's because you're responding to an entirely different issue. "Hotlinking" isn't linking to something, it's including a resource that is hosted elsewhere. It's putting <img src="https://concordDance.whatever/images/big_image.jpg"> on my website without asking you. Now if my site ends up on the front page of HN, that could cause a lot of traffic to your site, potentially overwhelming your server or increasing your hosting bill. It's not nice, and rightfully frowned upon.
In both cases the site loses bandwidth for no gain due to your actions.
If I tell the customer they can go next door to get a panini, I'm not stealing anything. Maybe that restaurant is packed right now and they'ed rather not have an extra customer, but there is a reasonable expectation that they would generally want customers or at least have a means of turning away unwanted customers otherwise.
On the other hand if I break into my neighbor's restaurant, make a panini, then bring it back to my restaurant to serve and make money off of, all without permission from the neighbor, I am most definitely stealing. Even if I doubt the neighbor will mind because he let me come over and make myself a panini once, I can't unilaterally act off that assumption.
Of course when someone doesn't want us to hotlink to their assets then don't do it.
[0] https://commons.wikimedia.org/wiki/Commons:Reusing_content_o...
Or maybe wikipedia is already mostly static.
also, I wonder if HN is inadvertently ddos'ing the ticket system ?
1. https://www.jwz.org NSFW!
I think browsers did drop the path from it at least.
I saw the described image but after I visited the site directly I couldn't see it any more when redirectly via hacker news. Saw it again when I opened an incognito tab.
other good ones were about:1994 and about:mozilla
hey, about:mozilla still works in firefox
https://www.google.com/search?q=firefox+gran+paradiso+robot&...
1. Is there a collective biological term for scrotum and it's contents that is not general like "genitals" is?
http://bash.org/?434593
Sorry to ruin the fun y’all but there’s images I won’t even mention that I can’t unsee and make me feel seriously ill when I do see them. I don’t want anyone else to feel that way without warning.
https://en.wikipedia.org/wiki/Shock_site
The Wikipedia page for https://en.wikipedia.org/wiki/Goatse.cx is text only and without any ascii art.
I'm amused that https://simple.wikipedia.org/wiki/Goatse.cx also exists.
I remember when I was about 15, before pop-up blockers were really a thing, someone sent me a link to that and it would keep opening popups with that image and you couldn't close all of them :-/
Sometimes people look back to the internet of the 90s with too rose-coloured glasses IMO.
I am personally most amused by #38659
Edit: this is a bit unfair, if its a specific app they should be convinced to cache just to avoid unfair resource usage, but hotlinking in general should not be seen as a problem
It's a waste of donors money if someone is using this image as some kind of "is this thing on" test using hacked computers...
But this goes beyond that - it's some blind check of internet connectivity for the app, and doesn't get shown to the user. We're pretty sure of that, given that with the amount of noise that task generated, if there was an app featuring that image at least one of the ~ 90M daily "views" would've been someone reading these posts.
Now, given we want to be nice, we didn't just blindly block the traffic, although making requests without user-agent is against our UA policy https://meta.wikimedia.org/wiki/User-Agent_policy
Things were done very differently back in the day. This problem would have been fixed real quick.
http://ascii.textfiles.com/archives/1011
I think it ended up being a sort of mobile-based botnet with a bizarre target, which luckily was deduced from some of the headers sent (they all had a random common header).
I'd bet that this is the flower of the week for them.
edit: queries?
https://pageviews.toolforge.org/mediaviews/?project=commons....
Simpler times.
For almost a week it remained the most requested image the post on which it appeared, the most popular.
It did make me uncomfortable, though. Fearing that my rankings would plummet or so.
My takeaway is nothing new: there are weirdo's online.
My blog did have the word "Peanutbutter" on one or two posts. And the word "sex" on another. Maybe at some point both words showed up close to one another when experimenting with some "random articles" sidebar or some "you may also like" list.
Per the comments, right now the top suspect seems to be the app "Josh" or another TikTok clone because of how traffic surged immediately after the TikTon ban:
https://twitter.com/bwaber/status/1358915338637873154
I suppose the contents of the image/"message" could change every day, but presumably that would be very obvious in the edit history of that file[0], unless Wikimedia Commons were suppressing the fact that the file is constantly changing. If they are part of the conspiracy, though, you'd think they would have taken down the task from Phabricator too.
[0] https://commons.wikimedia.org/wiki/File:AsterNovi-belgii-flo...
https://community.ntppool.org/t/recent-ntp-pool-traffic-incr...