The heart of that case is the cease & desist letter being treated as denying authorization.
On another note, I wonder how strongly this would hold up against an individual. e.g. suing someone that was permbanned from your site because they used a proxy to access the site
Beyond being difficult to find/sue the individual - and it being well beyond worth the costs of the lawsuit itself - I wonder if it would hold up in court.
I'll keep this bookmarked as a (rather empty) threat against trolls who are ban evading.
Denying authorization of the API, not of the website.
Important difference.
Especially given someone who uses web scraping doesn't necessarily mean they attempted to use the API, had an API key, or ever had the possibility of said non-existent API key being revoked.
Observing user requests (including deletes) has been a requirement of the API for as long time. The only thing that I'm surprised by is that it took Twitter this long to crack down. This doesn't mean these Tweets from public officials shouldn't or can't be preserved, it just means that doing so in this manner was always a misuse of Twitter's API.
Out of curiosity, what manner of saving them would you suggest instead? I'm not sure it's possible in an automated way without technically violating their TOS.
Saving tweets that have been deleted violates the rules you agree to in order to get an API key. Also, they no longer have API keys.
I think you could probably make an argument that it is fair use to save and display a tweet from a politician without violating anyone's copyright, but I'm not sure if it's possible to do it in an automated way without violating the TOS and possibly the law.
I think scraping is probably the simplest but the ethics are just as questionable. Any method you try is probably going to be met with roadblocks from Twitter because you are inherently trying to use the site in a way that is against their rules. Like you said, there may be a fair use argument to be made, but that won't stop Twitter from at least trying to make it as hard as possible for you.
Which is where that fair use argument will come in to play. That is something that can only play out in court. Twitter has no responsibility to bend to fair use until a court makes them.
They wouldn't be able to tell that it's automated though. You can probably do it with a single twitter account, without explicit API access. Just have your account follow all the politicians and screenshot the differences in your feed.
That's generally unenforceable unless you took affirmative action to agree to the terms. There's no proof the user even saw the terms if they're just scraping the site. Until you get a c&d letter, at which point it becomes harder to claim ignorance.
(I'm not a lawyer, IANAL) I don't get this. When did I officially agree to their ToS? I automatically agree on the first HTTP request? I think if I never had an account or never requested an API key then I didn't agree to it. Having said that there could be other legal issues to scraping (like copyright of posted tweets), but maybe ToS is not one of them.
That's pretty sad. I imagine there are plenty of API users that are doing actual nefarious things. This only comes of as being a back-bench political deal that Twitter made.
> "Imagine how nerve-racking — terrifying, even — tweeting would be if it was immutable and irrevocable?"
Yes, it would be like these ... um ... thingies that Twits never heard of like, oh, e-mail and Usenet.
And older thingies like writing letters. And, for example, having a "letter to the editor" published in a newspaper, which is then archived in a library.
Publishing should be nerve-wracking and terrifying.
You know, you can still buy calendars filled with George W. Bush gaffes. Should those be cease-and-desisted?
My comment comes from a simple observation that globally significant breaking news (that one isn't personally involved in) tends to happen on Twitter rather than people's inboxes. But thanks for the snide remark.
But I create API key for site: xyz.com where I use the stream API to get all tweets from users a-z
Then I push those to site zyx.com
There is no way for twitter to know what API generated those tweets. Especially if site xyz.com DOES obey the delete notifications (which also triggers site zyx.com to go into action in checking the deleted tweet)
They would be able to be shut down if they're using Twitter's own API keys - or iOS's built-in keys. These types of things can typically be easily found with strings or a proxy (Charles).
Wow, what a short-sighted and ill-conceived move on Twitter's part. I know the answer but I still have to ask- has Twitter used the internet before? Don't they know how it works?
The first thing many people do when they see a controversial tweet (especially from someone well known) is to take a screenshot. This changes nothing.
Many of these sites didn't originally spring up to serve the public interest. They're an offspring of something that started a couple years ago: Twitter bots that scrape for anyone's deleted messages and then highlight them.
Those services are extremely toxic. They meant that as friends of mine joined the service I'd have to warn them: don't delete things on Twitter, because there are asshole bots that will use deleted messages to fuck with you.
The public interest implications of some of these new political Twitter services are more straightforward, but that doesn't mean it's Twitter's obligation to sacrifice user experience for what is probably a marginal public interest benefit. Twitter is legitimately caught between the public interest and their own interests, and I'd have trouble blaming them either way.
Why do I think the public interest implications here are marginal? A bunch of reasons:
* Politicians are quickly going to learn how hostile Twitter is, and assign professional spindroids to their accounts (or to filter the messages that they themselves post).
* It's not like Twitter can prevent deleted tweets from leaking. They'll get screenshot anyways. This just keeps the delete-shaming phenomenon from being shoved in everyone's face.
On a previous thread about Twitter, someone pointed out that Twitter shouldn't do this because they should behave like infrastructure. But Twitter doesn't want to be infrastructure; the infrastructure market position is very bad for them.
Many of these sites didn't originally spring up to serve the public interest. They're an offspring of something that started a couple years ago: internet bots that scrape for anyone's deleted messages and then highlight them.
Those services are extremely toxic. They meant that as friends of mine joined the service I'd have to warn them: don't delete things on internet, because there are asshole bots that will use deleted messages to fuck with you.
The public interest implications of some of these new political internet services are more straightforward, but that doesn't mean it's internet's obligation to sacrifice user experience for what is probably a marginal public interest benefit. internet is legitimately caught between the public interest and their own interests, and I'd have trouble blaming them either way.
Why do I think the public interest implications here are marginal? A bunch of reasons:
* Politicians are quickly going to learn how hostile the internet is, and assign professional spindroids to their accounts (or to filter the messages that they themselves post).
* It's not like the internet can prevent deleted content from leaking. They'll get screenshot anyways. This just keeps the delete-shaming phenomenon from being shoved in everyone's face.
On a previous thread about the internet, someone pointed out that the internet shouldn't do this because they should behave like infrastructure. But the internet doesn't want to be infrastructure; the infrastructure market position is very bad for them.
This would be witty if Twitter were "the Internet", but of course they are not: they are a website on the Internet.
Your switch-the-words trope also stops making sense towards the end of the comment.
It's kind of an obnoxious way to make a point, too. Next time you snark like that, can you, like, capitalize the words you're substituting? I had to read this twice just to figure out what you were trying to say. Most of us just skim comments.
As a user, if I delete a tweet, I want it to be deleted. I don't want it to be on the internet anymore, because else I wouldn't have deleted it. Twitter management is defending their users.
If they wanted to create a general "politician public statement archive" that included deleted tweets, I think that would be acceptable.
Where they cross the line is in only doing deleted tweets, and marketing themselves that way. The entire purpose of their product is to undo a feature of someone else's product.
This story is being framed a certain way: "Twitter is a big corporation attacking the little guy in defense of evil politicians," and kids on Reddit are outraged. I don't think that's what's happening here at all. They're just defending the integrity of their product.
Think back thirty years. Imagine if ABC or CBS could demand that the New York Times not publish statements made by a politician on the nightly news or aggregate information about the political stances of a representative.
That's not what Twitter is doing. If that was their goal they'd be suing Gawker and TMZ.
The point isn't to stop deleted tweets from being disseminated at all - that's impossible and I'm sure they're smart enough to know that.
The sole purpose of sites like Politwoops was to undo a feature of Twitter's product. Not only that, but they were using Twitter's own API to accomplish that. Twitter's goal is to set the precedent that that's not ok.
Again, think back thirty years to a scenario where a politician misspeaks on the news. No network would ever deny a request for a tape or transcript of a news program based solely whether the politician wants his statement to disappear.
Everyone will agree that scraping and highlighting all deleted tweets is distasteful and slimy, but we're talking about politicians here. (Not that they're slimy, but that they exert an undue amount of power over the average bloke. Public interest, democracy, and all that.)
The analogy doesn't really make sense to me. Twitter isn't a news service, it's a communications platform, and Twitter users aren't necessarily politicians.
Twitter is a one to many communications platform, not a one to one platform.
The analogy holds because Twitter is 1:N, not 1:1. They are publishing information to a large amount of people that people use to educate and inform themselves.
And I already stated that doing this for every Twitter user is slimy. But retaining the record of how a politician has communicated with the public? That's in the public interest.
Anything you put on the internet is indeed immutable and irrevocable. This comment included. Hell, you don't even need to hit "submit" - the keystrokes and mouse movements - they are all captured and piped to many different analytics services in real time. It might not be surfaced on a publicly accessible site, but rest assured, it's in a database somewhere.
From TFA: [The] decision was the result of "thoughtful internal deliberation and close consideration of a number of factors" ...
They should address this truthfully on their blog. Letting something like this twist in the wind while their base users go apeshit is a surprising decision. They did this on a Friday- they've had all weekend to craft a statement.
That's always been the mystery of Twitter to me. The fact that it became a place where interesting people have (ostensibly) serious conversations is _incredible_ to me, given how amazingly ill-suited to that task it is. If it was just as successful but (only) full of people exhibiting drive-by "wit", it would make a lot more sense to me.
57 comments
[ 2.8 ms ] story [ 54.6 ms ] threadThere is need .. stealthy and robust .. tweet scraper.
On another note, I wonder how strongly this would hold up against an individual. e.g. suing someone that was permbanned from your site because they used a proxy to access the site
Beyond being difficult to find/sue the individual - and it being well beyond worth the costs of the lawsuit itself - I wonder if it would hold up in court.
I'll keep this bookmarked as a (rather empty) threat against trolls who are ban evading.
Important difference.
Especially given someone who uses web scraping doesn't necessarily mean they attempted to use the API, had an API key, or ever had the possibility of said non-existent API key being revoked.
I think you could probably make an argument that it is fair use to save and display a tweet from a politician without violating anyone's copyright, but I'm not sure if it's possible to do it in an automated way without violating the TOS and possibly the law.
Scraping a site that doesn't want to be scraped is legally dangerous in the US.
Scraping is against most sites (and Twitter)'s ToS.
Yes, it would be like these ... um ... thingies that Twits never heard of like, oh, e-mail and Usenet.
And older thingies like writing letters. And, for example, having a "letter to the editor" published in a newspaper, which is then archived in a library.
Publishing should be nerve-wracking and terrifying.
You know, you can still buy calendars filled with George W. Bush gaffes. Should those be cease-and-desisted?
Here, for "site" we can also substitute "archive". Archives don't have to process cancels or supersedes, and can ignore X-No-Archive headers.
Or, just run strings on the Twitter iOS app bundle... or just look at traffic in Charles...
But I create API key for site: xyz.com where I use the stream API to get all tweets from users a-z
Then I push those to site zyx.com
There is no way for twitter to know what API generated those tweets. Especially if site xyz.com DOES obey the delete notifications (which also triggers site zyx.com to go into action in checking the deleted tweet)
I don't see how this is a total non-story
Nothing new in this link.
The first thing many people do when they see a controversial tweet (especially from someone well known) is to take a screenshot. This changes nothing.
Many of these sites didn't originally spring up to serve the public interest. They're an offspring of something that started a couple years ago: Twitter bots that scrape for anyone's deleted messages and then highlight them.
Those services are extremely toxic. They meant that as friends of mine joined the service I'd have to warn them: don't delete things on Twitter, because there are asshole bots that will use deleted messages to fuck with you.
The public interest implications of some of these new political Twitter services are more straightforward, but that doesn't mean it's Twitter's obligation to sacrifice user experience for what is probably a marginal public interest benefit. Twitter is legitimately caught between the public interest and their own interests, and I'd have trouble blaming them either way.
Why do I think the public interest implications here are marginal? A bunch of reasons:
* Politicians are quickly going to learn how hostile Twitter is, and assign professional spindroids to their accounts (or to filter the messages that they themselves post).
* It's not like Twitter can prevent deleted tweets from leaking. They'll get screenshot anyways. This just keeps the delete-shaming phenomenon from being shoved in everyone's face.
On a previous thread about Twitter, someone pointed out that Twitter shouldn't do this because they should behave like infrastructure. But Twitter doesn't want to be infrastructure; the infrastructure market position is very bad for them.
Many of these sites didn't originally spring up to serve the public interest. They're an offspring of something that started a couple years ago: internet bots that scrape for anyone's deleted messages and then highlight them.
Those services are extremely toxic. They meant that as friends of mine joined the service I'd have to warn them: don't delete things on internet, because there are asshole bots that will use deleted messages to fuck with you.
The public interest implications of some of these new political internet services are more straightforward, but that doesn't mean it's internet's obligation to sacrifice user experience for what is probably a marginal public interest benefit. internet is legitimately caught between the public interest and their own interests, and I'd have trouble blaming them either way.
Why do I think the public interest implications here are marginal? A bunch of reasons:
* Politicians are quickly going to learn how hostile the internet is, and assign professional spindroids to their accounts (or to filter the messages that they themselves post).
* It's not like the internet can prevent deleted content from leaking. They'll get screenshot anyways. This just keeps the delete-shaming phenomenon from being shoved in everyone's face.
On a previous thread about the internet, someone pointed out that the internet shouldn't do this because they should behave like infrastructure. But the internet doesn't want to be infrastructure; the infrastructure market position is very bad for them.
Your switch-the-words trope also stops making sense towards the end of the comment.
It's kind of an obnoxious way to make a point, too. Next time you snark like that, can you, like, capitalize the words you're substituting? I had to read this twice just to figure out what you were trying to say. Most of us just skim comments.
Where they cross the line is in only doing deleted tweets, and marketing themselves that way. The entire purpose of their product is to undo a feature of someone else's product.
This story is being framed a certain way: "Twitter is a big corporation attacking the little guy in defense of evil politicians," and kids on Reddit are outraged. I don't think that's what's happening here at all. They're just defending the integrity of their product.
The point isn't to stop deleted tweets from being disseminated at all - that's impossible and I'm sure they're smart enough to know that.
The sole purpose of sites like Politwoops was to undo a feature of Twitter's product. Not only that, but they were using Twitter's own API to accomplish that. Twitter's goal is to set the precedent that that's not ok.
The analogy holds because Twitter is 1:N, not 1:1. They are publishing information to a large amount of people that people use to educate and inform themselves.
And I already stated that doing this for every Twitter user is slimy. But retaining the record of how a politician has communicated with the public? That's in the public interest.
They should address this truthfully on their blog. Letting something like this twist in the wind while their base users go apeshit is a surprising decision. They did this on a Friday- they've had all weekend to craft a statement.