Ask HN: manual or automated web scraping?

hi all, I've got something like 18000 websites I need to visit and extract some info from but every single one has a different organisation, there is no way I can make up rules unless I make 18000 of them or I spent weeks coming up with some sort of AI stuff.

At the same time I'm not an expert in the fields hence my question: would you got for manual scraping or you are aware of very sophisticated tools that I could try out? I'm aware of the well-known OSS ones (Weka, Scrapy, Beautiful Soup) but unless I'm missing something they fall into the "write as many rules as different websites you've got" category.

TA!