well, let's see what it once said at that URL... oh dear.
"A 21-year-old woman said Tarik Sansal raped her in the apartment's bathroom before 2 A.M. yesterday, the police said. Mr. Sansal was charged with first-degree forcible rape. The incident occurred in the apartment of Englebert Theuermann, a first secretary of the Austrian Mission to the United Nations..."
"Addendum: March 18, 2008, Tuesday. The Times did not report on subsequent developments. Tarik Sansal has a certificate of disposition by the Criminal Court of the City of New York, saying that the case against him was dismissed and sealed on June 5, 1997."
The incorrect reporting and update is newsworthy; but the article as it existed is problematic to have as a primary source on a search result.
There should be some way of annotating the contents are deprecated and referencing another hyperlink which explains the currently known facts. This link might be to a court case for an authoritative result of non-conviction / assumed innocence.
It's incorrect in the sense of the summary or what someone who only saw the initial version might draw as a conclusion. Further the update mentions this initial version was _historically_ (Edit: from sometime after publication in 1996 and not updated until 2008!!!) left online and indexed by search engines well past when followups should have updated it with conclusions about that case.
Hence it is 'incorrect' to present this article in a cursory search result, but it is correct to produce it as a link in relation to more specific searches, or better, to instead of this page, any designated URL link when a hit would point to this page.
That's how it's done in Norway and a bunch of other European countries, though the media usually don't even name convicted people unless there's major reasons/interest.
I dunno, it's tricky. Should it really be unlawful to report on the suspects identity if the government doesn't reveal the name of a suspect but the journalist(s) are able to figure it out by interviewing witnesses? Also tricky needle to thread habeas corpus while trying to protect the accused.
Yes, but secret trials don’t have a great history either. At its best, the media can prevent miscarriages of justice and make sure the government follows proper procedure.
There's a wide range between secret trials and not publishing peoples name in national media.
An approach used in many contexts is to make the information accessible but put limitations on publication whether permanently or only until a conviction is made, so that people with an interest in a case can investigate while recognising mistakes are made.
[another approach is various levels of barriers to make access inconvenient, e.g. forcing you to go to an office to look up records rather than making them easily searchable; or for tax information in Norway: the tax information is public, but unless you're media - who get a copy - your name will be accessible to the person you looked up]
Not necessarily. The crime scene was a diplomats residence, but the ddg results for the name are for some kind of New York businessman who might have been a guest. The dismissal of the case might have been an out-of-court agreement with lots of money involved.
Actions based on right to be forgotten don't affect what you're allowed to know or what you tell individuals, they affect what you're allowed to publish to the entire world. Much like publishing a photo of someone; sometimes you need permission.
This is a poor analogy. There is no particular reason to assign someone priveleges over the public facts of their life for example convictions trials misdeeds. In fact by definition no evil doer would ever give such permission.
Would you ask a rapist or their victim for permission to discuss the crime?
RTBF isn't ownership of information about self it is a privelege for villains to censor their victims and the general public.
The originator in the EU was a business man who wanted a mater of public record (some fine I believe) removed.
Does raise the q should states publish so much information they collect Swedish tax records, Names and Addresses of accused persons (lots of countries) publishing Mugshots like the US does.
> The originator in the EU was a business man who wanted a mater of public record (some fine I believe) removed.
IIRC it was a private person who wanted to remove an article about his house being foreclosed due to his debts. His request to remove the article was not granted since the article was lawful, but the court agreed that google should stop pointing to that article since it was like 15-20 years old and the man had paid of his dept and his house was no longer foreclosed.
What do you find villainous about that?
> Does raise the q should states publish so much information they collect Swedish tax records, Names and Addresses of accused persons (lots of countries) publishing Mugshots like the US does.
The right to be forgotten has nothing to do with publishing records of something. It is about search engines not pointing to something published in the past that can have an adverse effect on somebody today while the published information is irrelevant to anybody today ...
Publishing Names and Addresses of accused persons and their Mugshots is a great example. Publishing that information is relevant since the person is being accused at the time of publishing. A years later the person is found innocent. It is still true that the individual was accused but after being found innocent the information of them being accused isn't particularly relevant to anybody typing their name in a search engine but can have an adverse affect on the individual if he is looking for a job or whatever.
For that very reason a convicted criminal in my country (in the EU) can get a document saying that they are not a convicted criminal after they paid their dues to the society. The point is that people make mistakes and there is no good reason (in the very wast majority of cases) that those mistakes should follow them their whole lives. The point is rehabilitation not eternal punishment. That is a reason why the right to be forgotten exists.
> For that very reason a convicted criminal in my country (in the EU) can get a document saying that they are not a convicted criminal after they paid their dues to the society.
That’s an interesting phrasing. Even you (who seems to support the policy) describe the convicted criminal as a convicted criminal, yet they’re able to get a piece of paper that says they aren’t one.
Does everyone have to produce such a paper routinely in life, or is this a case of “hi, I’m @sokoloff and, even though you didn’t ask, I’d like you to read this paper which says I’m totally not a convicted criminal”?
“A, are you a convicted criminal?” “No.”
“B, are you a convicted criminal?” “No.”
“C, are you a convicted criminal?” “This piece of paper says I’m not.”
> That’s an interesting phrasing. Even you (who seems to support the policy) describe the convicted criminal as a convicted criminal, yet they’re able to get a piece of paper that says they aren’t one.
They can get that paper after they payed their dues to society (served their time).
> Does everyone have to produce such a paper routinely in life
Not routinely but I don't think anybody goes through their life without ever needing that paper. It is required for a vast variety of government programs and some jobs also require it. Working in IT I had to get it a few times when applying for jobs in private sector.
> It is still true that the individual was accused but after being found innocent the information of them being accused isn't particularly relevant to anybody typing their name in a search engine.
"Isn't particularly relevant" according to whom? Relevance in the eye of the beholder.
Presumably if google posted a list of everything they delisted it would defeat the purpose without such a list I cannot easily determine the nature of the content delisted.
The daily mail claims articles on Josef Fritzl a man who kept his daughter in a dungeon for 24 years and Tory MP Jonathan Djanogly who hired individuals to spy on fellow party members were effected but years after the complaint about them being delisted they are now findable on google again.
Google doesn't have a great history of doing the right thing with automated moderation. It's entirely likely that a script could easily deny a request to censor trivialities while covering the crimes of a pedophile without a human being in the loop.
More recently someone convinced google to delist the url
Their list of articles on abuse of the right to be forgotten for maximum irony.
Then there is Thomas Goolnik who is using RTBF to hide his efforts to hide his efforts. That is to say he is using RTBF to delist articles about him misusing the RTBF It seems the start of the chain of forgetting is Goolnik defrauding people out of a million dollars. Something that might be worth knowing if you were participating in any endeavor he was involved in.
Had the victim not been a unique UNION of {phone with battery + full video of scene with audio + harvard grad + well dressed} he would likely have ended up arrested, probably tried, and quite possibly convicted/pleaded, possibly even end up on a sexual predator listing.
Think of how many times this has happened. Now think of how star-crossed lucky the victim here was that he had irrefutable evidence to clear the false accusation.
> By his own account, Christian then said, "Look, if you're going to do what you want, I'm going to do what I want, but you're not going to like it,"
I hadn't heard that detail before. I can see how someone might legitimately interpret that as a threat.
Imagine being in her shoes: you're peacefully walking your dog in the park when a stranger confronts you, says "I'm going to do what I want, but you're not going to like it", and approaches your dog. Haven't we all seen enough videos to be concerned about what might happen next? From what I can tell, he seems like a good guy, and I don't think anything bad would have happened, but she couldn't know that at the time.
Which all reinforces the larger point: when the Internet reacts to a story, important details are buried and overlooked. Politicians and activists ignore the particulars of the case and shape it to fit their own agenda. The media sensationalizes the story for clicks. An outraged mob dismisses nuance as x-ism or y-phobia. Employers acquiesce to the demands of the mob simply to avoid becoming a target.
The right to be forgotten is important, because Internet justice is rarely just.
I thought the "right to be forgotten" is simply meant to prevent certain websites from appearing in search engine results. Isn't that precisely what robots.txt accomplishes? Does the right to be forgotten also compel website owners to censor their own content?
Accomplishes the same, but it's a different mechanism.
RTBF does not use robots.txt (and then require that it be respected) on the site hosting the thing to be forgotten, it's an exclusion on the search engine side.
... has a certificate of disposition by the Criminal Court of the City of New York, saying that the case against him was dismissed and sealed on June 5, 1997.
That’s how I read it too. Having the case sealed seems like an invitation to actually do some investigation. Instead the NYT played along and tried to bury the story (albeit in a technically ignorant way).
I am not a lawyer but I am fairly certain that New York State automatically seals criminal cases which are dismissed due to a positive finding for the defendant.
I've added a special exclusion to a robot.txt file for a specific article. It was some years ago while in college. The article in question was about the presentation of an assisting professor who had some kind of misunderstanding with the campus newspaper and therefore the article wasn't especially positive in tone. Couple of years later I was the sys admin of the newspaper website and a letter arrived in my university email. The professor had found that I'm responsible for the website and had sent me a tearful story about how this article is ruining her life, because it is the top Google result for her name, and how she had spent thousands of dollars on scammers who had promised to change that, and she was asking me to remove the piece. Long story short, I forwarded the case to the newspaper editor at the time and she agreed to let me add a line to the robot.txt.
Out of curiosity, I checked the student's newspaper website. It turns out that they've made redesign after my time and they've removed the robots.txt file. However, googling the name of the professor in question returns much more recent results and the article is hard to find. It turns out that Google's algorithm buries some stories over time. One more decade and noone will be able to find this part of her history unless they know what they are looking for.
It's incredibly hard to find any information that wasn't created this year.
Not sure if it's Google discriminating against old sites, or if the new sites just have such incredible levels of SEO-fu that a static, just-contains-what-you-want site has no hope of getting selected for results
Or it could also be that people are more often clicking on recent links or searching for "something something 2021". I know I usually give higher preference to links with more recent date attached as that just-contains-what-you-want site from 2010 will potentially be outdated by now. Of course it depends on subject of your search, but id imagine it's true more often than not in big scale.
This is the thing - for some things (e.g. searching for tech to use), you care about recency. But for something you absolutely don't. But the search engines magic seems to throw this all into one bucket and notice that people are more clicking the '2021' links. And the consequence of this is SEO optimization done by automatically updating all links to current year - I searched for something on 1st Jan and already got a bunch of '2021 comparisons'
It's a common content marketing tactic to update post titles to include "for 2021" a few weeks ahead of the new year. It's a tweak to get it ranking, as every other content marketer is doing the same.
Except when it isn't. I don't know if I'm uniquely bad at web search these days, but Google seems to do the exact opposite of what I want. I.e. when I need some solid information about a piece of technology or knowledge, it'll spam me with content marketing bullshit published last week. But when I need something recent - like a current opinion on some software alternatives, or installation instructions, all I get are results from 5+ years ago.
I don’t know if we’re both in that case and Google has simply moved away from how we learned to coerce it (and other search engines) over the years, our habits now yielding negative returns unbeknownst to us, but I find myself in a similar situation.
Month after month, year after year, search results get worse, and harder to sift through. At this point, google is mostly a way to find wiki articles despite typos (because Wikipedia’s search is bad and can’t handle typos).
Google has definitely changed in the past few years. I've noticed that it ignores quotes when it feels like it, and is very aggressive at "synonymous" substitutions (which in many cases aren't - e.g. I've seen it replace "FreeBSD" with "Linux"). It's making it harder and harder to make queries for something specific.
I thought I am the only one! Last week, I was trying to find something with the quotes and it completely ignore it. It took me a few queries to find what I am looking for.
That was mostly PSF's fault since they had the default documentation URLs setup to point at the Python 2 documentation for way longer than they should have...
Ha, me too, but then I've also been caught out on 'latest' (which I typically reflexively click on after opening) having some crucial difference from what we're using (12 vs. 13) too, so hard to win there.
I don't know off-hand if it's possible with the search (e.g. it might use a last-selected version cookie) but we should probably be using version-specific 'bangs'. `!pg12` or whatever, in my case.
So does Google, and it’s even worse there. In Google, the first page for "postgres window functions" is version 9.1, version 9.3, and 8 random tutorials. DDG has 9.1, 9.3, tutorial (the same as Google…), but then you at least get 12, 11, and then some more tutorials.
Google works for the masses. It assumes synonyms and aliases are great. For example, searching for phrases like debian <something>, often bring ubuntu help pages to the top results.
Worse if you are looking for Dave Smith. You'll get David Smith, because Dave=David, even though you know Dave never identifies as that.
Then add it fixing spelling errors. Which often are not. Rare term? Must be a spelling error!
Part of this, is because most people are non-precise, and also because Google wants voice input to work well. So there, their, they're are the same thing, but can also synonym to things like them, and they.
Point is, Google is trashy for any search not involving cats, or explosions. Quotes barely work, so the only way to mitigate this a bit, as google removed the + search modifier a decade ago, is always, use verbatim, under search tools.
And hope Google hasn't broken it that day. Which they do often.
That is one lucky professor -- how did she get granted an exception? There are tons of unlucky people who suffer as a result of bad/shoddy/vague reporting, how do they also get a carve-out?
In my case, nobody was actually malicious. It was a stupid situation caused by the stupid actions of both sides from what I've been able to discover. Nobody meant to cause harm but to tell their own story. I tried to find a compromise that would work for everyone and it worked this time.
This is now institutionalised as The Right To be Forgotten. Every search engine doing business in the EU has implemented it. Most probably only offer it to Europeans.
I believe it not only covers, but is intended to deal exactly with that - news pieces about individuals that are no longer relevant, but the individuals want to get out of search indices.
The rights to privacy and to be forgotten could be superseded in the cases of significant public interest. Especially the right to be forgotten is in regard to petty crimes, revenge porn or stupid posts done by minors. Right to be forgotten does not allow for censorship or rewriting history at the will of the subject of an article.
Something that Wikipedia reminded me, rtf is obligatory only in the EU and search engines are not obliged to comply with it on its international websites.
Similar thing here. I wrote a blog post about an online "contest" that was actually a bit of a scam. For years afterward I'd periodically get email from the person asking to take the blog post down because its prominence in search results was souring job prospects. I eventually relented, not because I cared about him but because he mentioned that he had started a family and it was affecting them as well. I didn't want to hurt innocent people. I didn't want to censor my blog either, but I did add a robots.txt so it wouldn't show up in search results.
Since then it seems he really has gotten it together, and even had a project show up on the first page here. So now I suppose the robots.txt entry doesn't matter much, but it's still there anyway.
Just for future reference: adding a URL to robots.txt will not necessarily exclude that web page from Google, especially if it has already been indexed.
To reliably exclude a URL from indexing, you have to serve a “no index” instruction with that URL, either in a meta tag or an HTTP header. And for this instruction to be read, the robot has to visit that page! So disallowing the URL in robots.txt can actually be counterproductive to de-indexing it.
Google also offers a tool specifically for removing URLs from their index in Search Console.
It's concerning someone working for the news doesn't understand why.
This is their power of ruining lives forever.
This is a new thing, it should be taught, it's not hard to understand.
I don't know why journalist think they deserve respect when these things are not fundamentally in their ethos.
There should be a better process than robot.txt and some news sites are doing better. Europe has brought in laws. But if journalist want to be thought of as more than writing blog spam, they need a better answer to this.
If you Google a job applicant's name and find several articles reporting that they have been accused of rape (with just a little note at the end that the charges have been dismissed), are you really going to give them a fair chance? I don't think anyone would look at them in an unbiased way, even if they tried to.
Why does it need to be a little note at the end. Change the title to so and so cleared of charges of rape if its so and yes I would give them a fair chance.
What is the alternative? I do not know anything about the particular case and make no claims about it but suppose the charges were dropped for political reasons or because the victim was intimidated? Shall we in such scenarios silence them entirely in order to protect a criminal?
the US Department of State (DoS) during 2012/2013 in its robots.txt[1] excluded around 9577 documents which leaked into archive.org (already pre Snowden). The robots.txt file now is OK but not sure if content is still on archive.
8<-----------8<-----------8<-----------8<-----------8<-----------
#!/bin/bash
snapshots="20120713050942 20121013154343 20121010165822 20120921054221 20130413152313 20130113162428"
# orig source http://state.gov/robots.txt but also on pastebin in case they delete it:
wget --output-document=robots.txt http://pastebin.com/raw.php?i=RE2tpyR3
for x in `echo $snapshots`
do
for i in `cat ./robots.txt|cut -d ' ' -f2 | tr -d '\15\32'`
do
if [ -e `basename $i` ]; then
echo "$i already fetched"
else
wget https://web.archive.org/web/$x/http://www.state.gov/documents/$i;
fi
done
done
This was one of the very rare cases where even wikileaks took down one article in the global intelligence files, after the state dpmt complained. A very high profile case. In that specific country everybody knew, what the diplomate of the other very specific country did.
I have a specific exclusion in my robots.txt file, and also a cron-scheduled grep of my logs to see if anything actually hits it. I don't care if they do or they don't, but it's a way for me to exclude specific bots that don't honor my robots.txt file.
Scathing Glassdoor review for his current company entitled "Just Another One of Tarik's Victims" among a sea of obviously fake ones (yes I know GD is bs)
Using robots.txt to avoid search engines indexing the page is the most stupid thing you can do. Not only it's not mandated by law that search engines have to follow the rules in the file, but also you are giving to the public a known file where you put all the URL that you don't want to be public. And everyone that wants to get some information on a site the first thing that goes to see is the robots file.
The correct thing would be to serve pages with the appropriate HTTP header to disable indexing. Of course search engines are still not obliged to follow the header, just as they are not obliged to follow the robots.txt file, but you are not leaking more information that you need.
Really, robots.txt file is only useful to reduce the load on the server by crawlers, it shouldn't be used as a protective measure!
102 comments
[ 2.2 ms ] story [ 155 ms ] thread"A 21-year-old woman said Tarik Sansal raped her in the apartment's bathroom before 2 A.M. yesterday, the police said. Mr. Sansal was charged with first-degree forcible rape. The incident occurred in the apartment of Englebert Theuermann, a first secretary of the Austrian Mission to the United Nations..."
"Addendum: March 18, 2008, Tuesday. The Times did not report on subsequent developments. Tarik Sansal has a certificate of disposition by the Criminal Court of the City of New York, saying that the case against him was dismissed and sealed on June 5, 1997."
There should be some way of annotating the contents are deprecated and referencing another hyperlink which explains the currently known facts. This link might be to a court case for an authoritative result of non-conviction / assumed innocence.
Hence it is 'incorrect' to present this article in a cursory search result, but it is correct to produce it as a link in relation to more specific searches, or better, to instead of this page, any designated URL link when a hit would point to this page.
The court of public opinion knows no presumption of innocence and can ruin the life of someone acquitted easily.
If they have to name the name they should wait until after conviction.
Also interesting, and quite sad: https://www.youtube.com/watch?v=UQSPBQjaWhI
For minor things, that’s not worth their time so they won’t name them.
Not a 100% practice, but I guess that’s what they teach in school.
An approach used in many contexts is to make the information accessible but put limitations on publication whether permanently or only until a conviction is made, so that people with an interest in a case can investigate while recognising mistakes are made.
[another approach is various levels of barriers to make access inconvenient, e.g. forcing you to go to an office to look up records rather than making them easily searchable; or for tax information in Norway: the tax information is public, but unless you're media - who get a copy - your name will be accessible to the person you looked up]
In the internet age it can heavily damage or ruin things like your job and dating prospects permanently.
Also his name plus his alleged crime gets an AP News article as the top hit.
Actions based on right to be forgotten don't affect what you're allowed to know or what you tell individuals, they affect what you're allowed to publish to the entire world. Much like publishing a photo of someone; sometimes you need permission.
Would you ask a rapist or their victim for permission to discuss the crime?
RTBF isn't ownership of information about self it is a privelege for villains to censor their victims and the general public.
This is repeated every time that the RTBF is mentioned yet none has ever given an example of RTBF being abused. Can you give an example?
Does raise the q should states publish so much information they collect Swedish tax records, Names and Addresses of accused persons (lots of countries) publishing Mugshots like the US does.
IIRC it was a private person who wanted to remove an article about his house being foreclosed due to his debts. His request to remove the article was not granted since the article was lawful, but the court agreed that google should stop pointing to that article since it was like 15-20 years old and the man had paid of his dept and his house was no longer foreclosed.
What do you find villainous about that?
> Does raise the q should states publish so much information they collect Swedish tax records, Names and Addresses of accused persons (lots of countries) publishing Mugshots like the US does.
The right to be forgotten has nothing to do with publishing records of something. It is about search engines not pointing to something published in the past that can have an adverse effect on somebody today while the published information is irrelevant to anybody today ...
Publishing Names and Addresses of accused persons and their Mugshots is a great example. Publishing that information is relevant since the person is being accused at the time of publishing. A years later the person is found innocent. It is still true that the individual was accused but after being found innocent the information of them being accused isn't particularly relevant to anybody typing their name in a search engine but can have an adverse affect on the individual if he is looking for a job or whatever.
For that very reason a convicted criminal in my country (in the EU) can get a document saying that they are not a convicted criminal after they paid their dues to the society. The point is that people make mistakes and there is no good reason (in the very wast majority of cases) that those mistakes should follow them their whole lives. The point is rehabilitation not eternal punishment. That is a reason why the right to be forgotten exists.
That’s an interesting phrasing. Even you (who seems to support the policy) describe the convicted criminal as a convicted criminal, yet they’re able to get a piece of paper that says they aren’t one.
Does everyone have to produce such a paper routinely in life, or is this a case of “hi, I’m @sokoloff and, even though you didn’t ask, I’d like you to read this paper which says I’m totally not a convicted criminal”?
“A, are you a convicted criminal?” “No.”
“B, are you a convicted criminal?” “No.”
“C, are you a convicted criminal?” “This piece of paper says I’m not.”
They can get that paper after they payed their dues to society (served their time).
> Does everyone have to produce such a paper routinely in life
Not routinely but I don't think anybody goes through their life without ever needing that paper. It is required for a vast variety of government programs and some jobs also require it. Working in IT I had to get it a few times when applying for jobs in private sector.
"Isn't particularly relevant" according to whom? Relevance in the eye of the beholder.
The daily mail claims articles on Josef Fritzl a man who kept his daughter in a dungeon for 24 years and Tory MP Jonathan Djanogly who hired individuals to spy on fellow party members were effected but years after the complaint about them being delisted they are now findable on google again.
Google doesn't have a great history of doing the right thing with automated moderation. It's entirely likely that a script could easily deny a request to censor trivialities while covering the crimes of a pedophile without a human being in the loop.
More recently someone convinced google to delist the url
https://www.techdirt.com/blog/?tag=right+to+be+forgotten
Their list of articles on abuse of the right to be forgotten for maximum irony.
Then there is Thomas Goolnik who is using RTBF to hide his efforts to hide his efforts. That is to say he is using RTBF to delist articles about him misusing the RTBF It seems the start of the chain of forgetting is Goolnik defrauding people out of a million dollars. Something that might be worth knowing if you were participating in any endeavor he was involved in.
Had the victim not been a unique UNION of {phone with battery + full video of scene with audio + harvard grad + well dressed} he would likely have ended up arrested, probably tried, and quite possibly convicted/pleaded, possibly even end up on a sexual predator listing.
Think of how many times this has happened. Now think of how star-crossed lucky the victim here was that he had irrefutable evidence to clear the false accusation.
I hadn't heard that detail before. I can see how someone might legitimately interpret that as a threat.
Imagine being in her shoes: you're peacefully walking your dog in the park when a stranger confronts you, says "I'm going to do what I want, but you're not going to like it", and approaches your dog. Haven't we all seen enough videos to be concerned about what might happen next? From what I can tell, he seems like a good guy, and I don't think anything bad would have happened, but she couldn't know that at the time.
Which all reinforces the larger point: when the Internet reacts to a story, important details are buried and overlooked. Politicians and activists ignore the particulars of the case and shape it to fit their own agenda. The media sensationalizes the story for clicks. An outraged mob dismisses nuance as x-ism or y-phobia. Employers acquiesce to the demands of the mob simply to avoid becoming a target.
The right to be forgotten is important, because Internet justice is rarely just.
Most Americans support right to have some personal info removed from online searches https://www.pewresearch.org/fact-tank/2020/01/27/most-americ...
they prefer to be able to remove information about themselves.
they strongly prefer you don't have the ability to remove information about someone else.
RTBF does not use robots.txt (and then require that it be respected) on the site hosting the thing to be forgotten, it's an exclusion on the search engine side.
that's pretty messed up.
Source: https://www.nycourts.gov/courthelp/criminal/sealedGoodResult...
edit: a word
Edit: newspaper -> newspaper editor
Old, to the point, websites hand-written in notepad.exe are rarely in the top 5-10, even when they have precisely the answer you’re looking for.
Not sure if it's Google discriminating against old sites, or if the new sites just have such incredible levels of SEO-fu that a static, just-contains-what-you-want site has no hope of getting selected for results
Google does this to make every tech worker not working for Google, and thus Google's competition, less productive.
</outlandish conspiracy theory>
Month after month, year after year, search results get worse, and harder to sift through. At this point, google is mostly a way to find wiki articles despite typos (because Wikipedia’s search is bad and can’t handle typos).
I don't know off-hand if it's possible with the search (e.g. it might use a last-selected version cookie) but we should probably be using version-specific 'bangs'. `!pg12` or whatever, in my case.
Information rots over time, decaying from true to false. Some facts are eternal, many aren't.
Worse if you are looking for Dave Smith. You'll get David Smith, because Dave=David, even though you know Dave never identifies as that.
Then add it fixing spelling errors. Which often are not. Rare term? Must be a spelling error!
Part of this, is because most people are non-precise, and also because Google wants voice input to work well. So there, their, they're are the same thing, but can also synonym to things like them, and they.
Point is, Google is trashy for any search not involving cats, or explosions. Quotes barely work, so the only way to mitigate this a bit, as google removed the + search modifier a decade ago, is always, use verbatim, under search tools.
And hope Google hasn't broken it that day. Which they do often.
The newspaper doesn't have to do anything.
Something that Wikipedia reminded me, rtf is obligatory only in the EU and search engines are not obliged to comply with it on its international websites.
https://en.wikipedia.org/wiki/Right_to_be_forgotten
Since then it seems he really has gotten it together, and even had a project show up on the first page here. So now I suppose the robots.txt entry doesn't matter much, but it's still there anyway.
To reliably exclude a URL from indexing, you have to serve a “no index” instruction with that URL, either in a meta tag or an HTTP header. And for this instruction to be read, the robot has to visit that page! So disallowing the URL in robots.txt can actually be counterproductive to de-indexing it.
Google also offers a tool specifically for removing URLs from their index in Search Console.
https://www.wired.com/robots.txt
It's concerning someone working for the news doesn't understand why.
This is their power of ruining lives forever.
This is a new thing, it should be taught, it's not hard to understand.
I don't know why journalist think they deserve respect when these things are not fundamentally in their ethos.
There should be a better process than robot.txt and some news sites are doing better. Europe has brought in laws. But if journalist want to be thought of as more than writing blog spam, they need a better answer to this.
What is the alternative? I do not know anything about the particular case and make no claims about it but suppose the charges were dropped for political reasons or because the victim was intimidated? Shall we in such scenarios silence them entirely in order to protect a criminal?
https://www.wnycstudios.org/podcasts/radiolab/articles/radio...
If you can’t get NYT to remove it, maybe you “know someone” who can.
robots.txt version from 2017: https://www.robots-viewer.com/robots/checksum/8029662cfb040c...
[1] https://pastebin.com/raw/RE2tpyR3
The correct thing would be to serve pages with the appropriate HTTP header to disable indexing. Of course search engines are still not obliged to follow the header, just as they are not obliged to follow the robots.txt file, but you are not leaking more information that you need.
Really, robots.txt file is only useful to reduce the load on the server by crawlers, it shouldn't be used as a protective measure!