HN Explorer

Ask HN: What is nitter and why does it still work?

54 points by blueridge ↗ HN
You can't access X without signing in, then there are the rate limits if you are signed in, but these things are not an issue with nitter. I went from checking Twitter as a logged out user every few days, to not using it at all after the access changes, to casually using nitter to check in on a few accounts I like.

https://nitter.net/search

21 comments

[ 3.0 ms ] story [ 246 ms ] thread
brucethemoose2
Historically it worked by using Twitter's undocumented internal API, but I have no idea what it does these days.

Its not even getting rate limited anymore.

linusg789
https://github.com/zedeus/nitter/

I think it uses the same functionality as the guest account feature in the Android app as of now.

loveparade
If that's true, isn't it just a matter of time before it gets shut off? I don't really see a way to keep this working without somehow crowdsourcing accessing X. Even with an automated way to create fake accounts with phone verification, it would eventually be detected or guarded against.
toomuchtodo
That’s how archive.today works with news publication site accounts, will be fine if you have to make round robin requests with a bucket of X accounts.
loveparade
I would disagree here, I think it's different for several reasons:

- The volume and velocity of data and API requests needed for X is much larger than scraping the occasional news article. You'd need a huge number of accounts to make it work.

- X has the technology to reliably detect such accounts unless they are incredibly well "hidden" by e.g. crowsourcing requests through regular accounts with normal browsing behavior

- X has the incentive to reliably detect such accounts as it has shown in the past by shutting off data access. It seems to care much more about this than most news websites

I believe the only reason things like Nitter still work is that it's not yet big enough for X to care about and invest resources into guarding against it, especially with all the other stuff going on at X right now. But it's only a matter of time before that changes.

toomuchtodo
I can tell you that a project is already doing this today for archiving with no signs of mitigations (request denials, poisoning the responses, etc). It’ll likely eventually end up in the usual places when you want to see something from way back in the day.
nitwit005
> But it's only a matter of time before that changes.

If anyone at the helm is paying attention, which seems iffy these days.

midoridensha
>- X has the technology to reliably detect such accounts unless they are incredibly well "hidden"

Has, or had? With massive layoffs, they might not have the staff or know-how left to do this, or the time to implement it.

>- X has the incentive to reliably detect such accounts as it has shown in the past by shutting off data access. It seems to care much more about this than most news websites

OK, but still wanting to do something, and actually doing it, are two different things. I'm not convinced this hollowed-out shell of a company still has such ability.

>especially with all the other stuff going on at X right now. But it's only a matter of time before that changes.

This seems to assume X is going to continue to survive and in fact thrive; I don't think that's a good assumption.

da9
They arent scrapping the whole site. The entire site is 99% spam. Its scrap on request, if its not already cached.
loveparade
Sure, but that doesn't make a difference. Even the top 1% of the most requested timelines and tweets are huge amounts of requests. That's many many orders of magnitude more data then new articles and requires a large number of accounts.
pimlottc
That should be “scrape”/“scraping”, unless you mean they should be throwing the site away :)
rsynnott
I mean, possibly? The loginwall thing is a fairly obviously silly strategy which they may not be excited about doubling down on, though (I would expect it's fairly likely that the loginwall gets reversed at some point; they've already stepped it down so that non-logged-in users can see tweets).

But honestly, at this point, _everything_ about twitter is up in the air. Who knows what state it'll be in in a year.

32gbsd
No clue how it works nowadays
JCharante
The author made a post bragging about how easy it was to circumvent rate limits since X doesn’t even limit by IP address, so they had 1 address making requests for like 1000 accounts without any problems.
supriyo-biswas
When I checked this a few days ago, Nitter wasn’t returning the latest data, so aggressive caching would be one of the tricks they’re using.
nyanpasu64
I self-host a LAN-only Nitter instance for personal use. In the last few months of Elon trying to lock out third-party clients, it's gone from automatically acquiring tokens, to tweet access breaking and coming back and eventually breaking again, then only showing "top tweets" for users rather than full post history, to having to log into Twitter in a browser and extract some cookies/headers using Developer Tools (and failing to show user post history because they tried using RSS feeds that got pulled). The latest development is the use of "guest tokens" (https://github.com/zedeus/nitter/pull/985) that you have to create and install by hand (running a Node script) because Nitter hasn't automated the process yet. This works well for viewing all tweets, but hasn't been merged to the master branch yet (but nitter.net and the AUR nitter-git package are already using it).