I did this as a fun exercise using Python and its Scrapy library.

It was a lot of fun! I recommend doing this.