Whenever I visit someone's Twitter profile without being logged in, I see an assortment of years old tweets and nothing current. Do other people experience this too? I assumed that I was being A/B tested with some limited view after the forced login modal stopped appearing.
Yes, it happens to all profiles I visit and I can rule out the pins/highlights feature. There's a weatherman in Texas who is a prolific tweeter and his page shows a random assortment of years old retweets (~2015), some tweets from last month, and no current weather forecasts. They don't change between page visits and none of them are in chronological order. What do you see when you visit https://twitter.com/wfaaweather ?
> Whenever I visit someone's Twitter profile without being logged in, I see an assortment of years old tweets and nothing current. Do other people experience this too?
Yes. Views of logged-out profiles display statuses sorted by most to least likes, rather than chronologically. It's incredibly dumb, and makes many accounts (like government announcement accounts - @NWS et al) useless while logged out.
What a horrible decision. With this change I get the sense that the user has a dead account, or the page is broken. Definitely not some compelling "top hits" compilation that would make me want to sign up for 'X'.
Some people like(d) to compare Musk to Jobs; but when it comes to Product instincts - they couldn't be further apart. Musk is the anti-Steve-Jobs, his ability to make the worst product decisions is preternatural.
Up to the lawyers and the courts. And I can definitely see how it could be interpreted to be a legitimate "bots are not authorized to view this site" directive since robots.txt is a standard which is respected across the industry.
If you want to completely ignore context, then sure. I got a chuckle. But I think ad absurdism didn’t work here.
Also, you deleted your comment conveying disdain over the Twitter to X rebrand. Considering rebranding is very common in mergers and acquisitions, would you say you feel strongly about it, and if so, why do you feel that way?
I deleted it because I found this comment was crap. It did not bring anything useful, and was not funny neither. And would have risked bringing sterile discussion. It was just not at the level of what I expect from myself. Sorry if you tried to reply and failed as a consequence.
> would you say you feel strongly about it
No. There are so many things that are more meaningful and enjoyable to focus on in life that spending emotion on some rebranding is not worth it. I'm not even a user of the thing.
The rename is stupid though, even more so than my deleted comment, no doubt about it. ;-)
Originally, the ninth circuit ruled that web scraping was allowed. This was overturned by a Supreme Court decision. It was ultimately found that HiQ was in violation of the User Agreement and they settled with LinkedIn.
> LinkedIn petitioned the Supreme Court to review the Ninth Circuit's decision.[9] In an order on June 14, 2021,[10] the Supreme Court vacated the Ninth Circuit's decision on the basis of their ruling on CFAA the week prior in Van Buren v. United States, which had ruled that the "exceeds authorized access" of CFAA only applies when an individual has valid access to a system but accesses parts of a system they are not intended to access.[4] The case was remanded to the Ninth Circuit for further review under Van Buren.[11]
> In a second ruling in April 2022 the Ninth Circuit affirmed its decision.[5][6]
The final ruling on this case was in November 2022[1]. The April 2022 decision was overturned in August 2022. It came back up in November 2022, where it was decided that HiQ has violated the User Agreement and had to pay damaged to LinkedIn.
HiQ no longer exists, so I think the case is definitely done. It doesn't seem to have set any firm precedent about whether web scraping violates the CFAA, but considering HiQ ultimately had to pay damages, I don't think its a resounding win for web scrapers.
It says they remanded it and the court it was remanded to affirmed its decision. "Its" is ambiguous, but it refers to the court it was remanded to, this is the source they reference:
> In its second ruling on Monday, the Ninth Circuit reaffirmed its original decision and found that scraping data that is publicly accessible on the internet is not a violation of the Computer Fraud and Abuse Act, or CFAA, which governs what constitutes computer hacking under U.S. law.
Can you compare a robots.txt file on a website to a sign on a physical door that says "authorized admittance only"? Both the website and the building are open to the public, but there are designated areas for authorized visitors only.
much like a physical business having the right to refuse service, a website should also be able to refuse serving data to users that have abused the service. a bartender says you've had too many, so no more service. websites can rate limit as well. we recently saw Musk test this concept.
Where I live, you can't really legally refuse serving an individual without a strong reason. And if you did, you'd better be able to prove that you didn't do it for discriminatory reasons.
Here, bartenders can say you've had too many, they are supposed to stop selling you alcohol if they think it could be dangerous for you or other people. This is one of the cases where not only the business can refuse to serve you, it also has to.
Now, I'm not a lawyer but I wouldn't expect robots.txt to be legally enforceable.
Of course, in any case, the website can totally refuse to fulfill a request, rate limit, etc. I'd say this is not comparable to the bartender or a business refusing to serve you though. Responding to an HTTP request is not offering a commercial service. And if it is, whatever legal document that promises you the service can totally specify that they won't promise to answer your robot.
If you're being an nuisance and causing the other patrons to have a less than ideal experience, absolutely, you should be asked to leave. If your site is constantly getting scraped and your service is affected due to it which causes other users to have slowed responses, you should absolutely be able to refuse requests from known knuckleheads.
To this day, bars have a list of "refuse entry". Sporting teams have life time bans to hooligans. Sure, the ban hammer should be slow to swing, but it shouldn't a never allowed to swing situation.
robots.txt is such a shitty implementation, it's like putting the "authorized admittance only" sign on the back of the door you're not allowed to traverse.
Were it a standard, it wouldn't be so easily neglected without consequence. In practice it's an arbitrary playground rule, like declaring the floor is actually lava. Unless you're directly involved with coming up with these "standards," you can go your entire career in total ignorance of all of them.
robots.txt does not say "authorized admittance only". It says "I'd like you to abide by these rules pretty please".
It's there for entities willing to (try to) respect website owners wishes and preferences.
It's useful when you need accessing stuff in an automated way, while staying on good terms.
By downloading robots.txt, the entity is asking the website owners their preferences regarding crawling. It's polite to do so. robots.txt can also offer useful guidance ("yeah, don't go there... [it's useless, too costly, etc], but you can go there...").
It's not a guarantee. And entities can totally decide that the request is unreasonable. Their call. The risk is being shamed or blocked.
Personally, I find it unreasonable / harmful to allow Google and deny everything else. If I built a crawler for a search engine, I would probably not follow this. And Twitter might find out and block me, of course.
robots.txt is not a security measure / measure against abuse. This is what IP ban and other kinds of blocking are for. But it's still useful. Not flawless, but useful.
I am not a lawyer but I am pretty sure a website protected with a password and/or firewall means something different from a robots.txt that is nothing more than a convention.
Yes, the robots.txt file is for agents that cooperate, but voluntary cooperation is actually pretty important. Put that line in your robots.txt and disappear from Google.
(Twitter specifically allows Google earlier in its robots.txt file.)
Not sure it applies to twitter specifically, but if you are scraping websites you should check robots.txt. The intended purpose is to direct you to where the content actually is, so you're not wasting resources spidering through contentless pages, just because they're linked from somewhere.
B. "I'm going to scrape Twitter data to do X, Y, Z"
A is far more likely to care about robots.txt because there's enough left of the web to complete their mission. B will not care if Twitter has a robots.txt file.
If you open a bakery in Europe, do you have to buy flour from every single farmer on the continent? Or can you "treat parties in unequal ways" and choose who you do business with?
You don't need to opt-in to terms of services. The service provider can simply refuse to service you if you violate the ToS. Violating ToS is not a crime.
However, if you actively try to circumvent this refusal of the service provider, this could very well be a crime, depending on what you are doing.
>From what I've heard many of those lawsuits have let him settle for less, including the rent
Of course, that's how settlements work. Elon literally has more wealth to bully people than anyone else on the planet, and is willing to do things out of spite.
Won't this robots.txt update stop major search engines from indexing Twitter? Seems like a great way to kill revenue as Twitter vanishes from search results.
IIRC there have been supreme court rulings about the legality of web scraping but they were all publicallu facing services like Amazon or other e-commerce sites.
Twitter is locked behind a login wall now so the rules in theory might be different.
Of course, bots can still get in through the human door.
1-2 years ago they used to automatically suspend new accounts without a phone number within 1h after registration; after that you had to go through a manual account recovery process. Did something change here?
Most people would not consider it easy to acquire burner phones in bulk (especially with the restriction they've never been used to sign up for Twitter).
Maybe to register, but in my experience (trying to set up an official account for a project) they are quickly locked the moment you try to actually do anything with them, requiring a email link followed by a unique multi-step captcha.
I hope services like HN and others rapidly go through their database of links that point to Twitter and mirror the content off into actually-public archives. Because it's only a matter of time before work-arounds like Nitter and others go belly-up.
There's unfortunately massive amount of content produced and reproduced by people who imagined they were doing so on a public forum. But it's increasingly going behind a wall of 403s and obfuscation.
For a few days in July non-Twitter members couldn't see anything there. Now they can only see the primary post and not the threads. That's Musk's right to do that, but the right thing to do is for people to clue in that this isn't a public resource, and if they want their content to be broadly available they'll need to find other places to put it. And not just for posterity, but for the present as well.
One thing I don't understand is how the advertisers who nominally fund Twitter (or did) are fine with this. Seems like deliberately kneecapping the reach and impressions.
If requests from who? or what? Account? IP? Because no one is doing it from one account or one IP.
> There are plenty of solutions for these types of problems.
Actually not a lot, it's a really hard problem to solve, unless you're referring to unsophisticated scrapers that have no idea what they're doing and are following some online tutorials on how to scrape Twitter. Out of desperation, they even implemented blanket bans on entire legitimate mobile carriers in some EU countries, but it didn't have a lot of impact on the business of Twitter scraping.
Yes, it is a hard problem. Legal justice is also a hard problem. My above point is that twitter can unilaterally take steps to enforce their policy, i.e. they do not need to lean on legal mechanisms.
There's another strategy possible here - technical blocking will prevent almost all scraping (proxies are damn expensive for hobbyists and hackers!). For people who go around these limitations, it will speak to their intent to bypass access limits and they can be taken down in a court.
Non-business-entities of course may escape judgement if they cannot be found or are in a place that doesn't enforce these kinds of laws. However, I bet Twitter's bigger fear is the US and EU commercial entities and generative AI - limiting access and then going after them in court will stop a lot of commercial enterprises.
Did you finish reading the page you linked? After the Supreme Court overruled the ninth circuit, HiQ was found to be in violation of the user agreement. They had to settle and pay $500k to LinkedIn.
I'm very familiar with the case, HiQ was found in violation because they also created bot accounts from which they consented into the user agreement. LinkedIn lost based on merits of the scraping of public data.
"its data" - nothing on Twitter, except the pathetic memes from a billionaire wannabe edgelord should "belong" to Twitter. It's all user-generated content.
I know it'll never happen because we live in the reality where everything must be awful all the time, but I'll continue dreaming of a day where large internet platforms are regarded and regulated as public utilities.
I've come to feel that regulating a platform like Twitter as a public utility could be a well intentioned but unfathomably evil act.
Tweeting is the ultimate example of an activity that's designed to be engaging but is otherwise awful. I think it may be the very nature of popularity-driven ultra-short-form content that it encourages online mobs, cancel culture, dunks and hot takes.
If there's even a chance this is true then it would be criminal to calcify this medium in its currently known form as a permanent pillar of our public square - which regulating it as a public utility would do. We could be inflicting suffering on future generations that might have escaped it.
No. It's better to hope that we discover better ways of communicating, and tweeting becomes irrelevant and dies.
> I think it may be the very nature of popularity-driven ultra-short-form content that it encourages online mobs, cancel culture, dunks and hot takes.
What "cancel culture"? Who got "cancelled" for real? Literally everyone of those whining about "cancel culture" who hasn't actually broken laws or is being suspected having done so (Tate) is still running around and making tons of money whining around to everyone.
Funnily enough people on the right are largely immune to cancel culture because they have their own support structures. Meanwhile innocent people on the left get their lives ruined by cancel culture twitter zombies that think harassing and stalking people makes them righteous.
105 comments
[ 2.1 ms ] story [ 193 ms ] thread[Edit] I should add that nobody has paid me for accessing my site that has a ToS/AUP requiring everyone to pay me 10% of their monthly income.
Yes. Views of logged-out profiles display statuses sorted by most to least likes, rather than chronologically. It's incredibly dumb, and makes many accounts (like government announcement accounts - @NWS et al) useless while logged out.
What a disaster.
Some people like(d) to compare Musk to Jobs; but when it comes to Product instincts - they couldn't be further apart. Musk is the anti-Steve-Jobs, his ability to make the worst product decisions is preternatural.
I deleted my Twitter account before all that, all too often I cannot even see Twitter deeplinks anymore.
edit: even Twitter don't really have much faith in those rules:
"might possibly". lol. Really sounds like "probably won't work but can't hurt, eh?"Up to the lawyers and the courts. And I can definitely see how it could be interpreted to be a legitimate "bots are not authorized to view this site" directive since robots.txt is a standard which is respected across the industry.
I sure would hope it would not cut it in courts.
Also, you deleted your comment conveying disdain over the Twitter to X rebrand. Considering rebranding is very common in mergers and acquisitions, would you say you feel strongly about it, and if so, why do you feel that way?
I deleted it because I found this comment was crap. It did not bring anything useful, and was not funny neither. And would have risked bringing sterile discussion. It was just not at the level of what I expect from myself. Sorry if you tried to reply and failed as a consequence.
> would you say you feel strongly about it
No. There are so many things that are more meaningful and enjoyable to focus on in life that spending emotion on some rebranding is not worth it. I'm not even a user of the thing.
The rename is stupid though, even more so than my deleted comment, no doubt about it. ;-)
Those are generally accepted as legal and enforceable. Though enforcing a pants color might be hard.
Originally, the ninth circuit ruled that web scraping was allowed. This was overturned by a Supreme Court decision. It was ultimately found that HiQ was in violation of the User Agreement and they settled with LinkedIn.
> In a second ruling in April 2022 the Ninth Circuit affirmed its decision.[5][6]
Is this not saying the opposite?
HiQ no longer exists, so I think the case is definitely done. It doesn't seem to have set any firm precedent about whether web scraping violates the CFAA, but considering HiQ ultimately had to pay damages, I don't think its a resounding win for web scrapers.
[1] https://www.natlawreview.com/article/court-finds-hiq-breache...
For an interesting breakdown including from a lawyer on this very site.
At least, that's the last ruling noted in the wiki article.
The SC also basically ruled that that HiQ did not have the right to scrape LinkedIn's website.
> In its second ruling on Monday, the Ninth Circuit reaffirmed its original decision and found that scraping data that is publicly accessible on the internet is not a violation of the Computer Fraud and Abuse Act, or CFAA, which governs what constitutes computer hacking under U.S. law.
https://techcrunch.com/2022/04/18/web-scraping-legal-court/
Here, bartenders can say you've had too many, they are supposed to stop selling you alcohol if they think it could be dangerous for you or other people. This is one of the cases where not only the business can refuse to serve you, it also has to.
Now, I'm not a lawyer but I wouldn't expect robots.txt to be legally enforceable.
Of course, in any case, the website can totally refuse to fulfill a request, rate limit, etc. I'd say this is not comparable to the bartender or a business refusing to serve you though. Responding to an HTTP request is not offering a commercial service. And if it is, whatever legal document that promises you the service can totally specify that they won't promise to answer your robot.
To this day, bars have a list of "refuse entry". Sporting teams have life time bans to hooligans. Sure, the ban hammer should be slow to swing, but it shouldn't a never allowed to swing situation.
Were it a standard, it wouldn't be so easily neglected without consequence. In practice it's an arbitrary playground rule, like declaring the floor is actually lava. Unless you're directly involved with coming up with these "standards," you can go your entire career in total ignorance of all of them.
It's there for entities willing to (try to) respect website owners wishes and preferences.
It's useful when you need accessing stuff in an automated way, while staying on good terms.
By downloading robots.txt, the entity is asking the website owners their preferences regarding crawling. It's polite to do so. robots.txt can also offer useful guidance ("yeah, don't go there... [it's useless, too costly, etc], but you can go there...").
It's not a guarantee. And entities can totally decide that the request is unreasonable. Their call. The risk is being shamed or blocked.
Personally, I find it unreasonable / harmful to allow Google and deny everything else. If I built a crawler for a search engine, I would probably not follow this. And Twitter might find out and block me, of course.
robots.txt is not a security measure / measure against abuse. This is what IP ban and other kinds of blocking are for. But it's still useful. Not flawless, but useful.
(Twitter specifically allows Google earlier in its robots.txt file.)
A. "I'm going to scrape the web to do X, Y, Z"
and
B. "I'm going to scrape Twitter data to do X, Y, Z"
A is far more likely to care about robots.txt because there's enough left of the web to complete their mission. B will not care if Twitter has a robots.txt file.
I wonder about Bing though, since they do seem to have Twitter results.
Sounds like discrimination.
Why Google can scrape data, and I as individual can't?
Sorry I'm from Europe, so I'm not that much into "what corporate wants, it gets" kind of thing.
However, if you actively try to circumvent this refusal of the service provider, this could very well be a crime, depending on what you are doing.
https://arstechnica.com/tech-policy/2023/09/musks-unpaid-bil...
Of course, that's how settlements work. Elon literally has more wealth to bully people than anyone else on the planet, and is willing to do things out of spite.
"Googlebot" might as well be the "Mozilla/5.0" of search index bot user agents.
Twitter is locked behind a login wall now so the rules in theory might be different.
Of course, bots can still get in through the human door.
1-2 years ago they used to automatically suspend new accounts without a phone number within 1h after registration; after that you had to go through a manual account recovery process. Did something change here?
There's unfortunately massive amount of content produced and reproduced by people who imagined they were doing so on a public forum. But it's increasingly going behind a wall of 403s and obfuscation.
For a few days in July non-Twitter members couldn't see anything there. Now they can only see the primary post and not the threads. That's Musk's right to do that, but the right thing to do is for people to clue in that this isn't a public resource, and if they want their content to be broadly available they'll need to find other places to put it. And not just for posterity, but for the present as well.
One thing I don't understand is how the advertisers who nominally fund Twitter (or did) are fine with this. Seems like deliberately kneecapping the reach and impressions.
https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn
What you mean by that? Currently it's not enforceable by any technical means, as I can scrape Twitter without any limitations.
If requests from who? or what? Account? IP? Because no one is doing it from one account or one IP.
> There are plenty of solutions for these types of problems.
Actually not a lot, it's a really hard problem to solve, unless you're referring to unsophisticated scrapers that have no idea what they're doing and are following some online tutorials on how to scrape Twitter. Out of desperation, they even implemented blanket bans on entire legitimate mobile carriers in some EU countries, but it didn't have a lot of impact on the business of Twitter scraping.
Yes, it is a hard problem. Legal justice is also a hard problem. My above point is that twitter can unilaterally take steps to enforce their policy, i.e. they do not need to lean on legal mechanisms.
Non-business-entities of course may escape judgement if they cannot be found or are in a place that doesn't enforce these kinds of laws. However, I bet Twitter's bigger fear is the US and EU commercial entities and generative AI - limiting access and then going after them in court will stop a lot of commercial enterprises.
https://www.natlawreview.com/article/hiq-and-linkedin-reach-...
https://www.natlawreview.com/article/court-finds-hiq-breache...
I know it'll never happen because we live in the reality where everything must be awful all the time, but I'll continue dreaming of a day where large internet platforms are regarded and regulated as public utilities.
Tweeting is the ultimate example of an activity that's designed to be engaging but is otherwise awful. I think it may be the very nature of popularity-driven ultra-short-form content that it encourages online mobs, cancel culture, dunks and hot takes.
If there's even a chance this is true then it would be criminal to calcify this medium in its currently known form as a permanent pillar of our public square - which regulating it as a public utility would do. We could be inflicting suffering on future generations that might have escaped it.
No. It's better to hope that we discover better ways of communicating, and tweeting becomes irrelevant and dies.
What "cancel culture"? Who got "cancelled" for real? Literally everyone of those whining about "cancel culture" who hasn't actually broken laws or is being suspected having done so (Tate) is still running around and making tons of money whining around to everyone.