Ask HN: How do I scrape my tweets without using Twitter's own tool?

4 points by matteocapucci 3y ago ↗ HN

Given the state of the Twitter workforce I suspect my request for data wouldn't be fulfilled anytime soon. Yet, there must be a way to scrape my tweet history just using the APIs. Does anyone know of any good (and ideally free) online service I can use for this?

2 comments

[ 2.8 ms ] story [ 12.4 ms ] thread

suramya_tomar 3y ago ↗

You can use Tweepy (Python Package) to query their API for the data. You will need a Twitter Developer Account but the setup is quite simple. You can use the instructions at: https://towardsdatascience.com/how-to-access-data-from-the-t... to set it up.

I wanted a client to export my followers, Following and Lists (Members) as the official export didn't have it. Used Tweepy to create a script in a couple of hours to pull this data. It should it fairly simple to modify it to pull the tweets as well if you are interested. Or I can take a look and add it when I get some free time.

https://github.com/suramyatomar/TwitterExtract

DantesTravel 3y ago ↗

I've recently discovered on a substack about web scraping the existence of anti detect browsers. Using Selenium or Playwright you can automate these browsers, each can have hundreds of isolated profiles, with different fingerprints and cookies. If you add also a proxy layer, you should be ok.