Hi all! Thank you very much for trying out Crawlcrawler.
I lauched the site yesterday and had anticipated that problems might arise in the crawl process because of a naive HTML parser that cannot run JS and indeed my crawler is unable to extract anything meaningful from many sites out there, imdb.com and twitter to name a couple. I don't plan on ever running JS because I think client-side JS is a mistake all together. Must-have UI-related javascript should become HTML elements</rant>. Anyway, I'm working on fall-back parsing methods that I think will help extract text even from heavily JS-driven sites. That's coming soon.
A for me entirely new problem has arisen, though, thanks to those of you who submit urls such as these:
First of all, because of the double slashes, server returns 404. But how in the lords name am I supposed to know to discard some query params but keep others? It seems an unsolvable problem to me. Please advise.
Edit: On the one hand it seems like it's the web server's job to tell me which of these resources I should use as the canonical one. On the other hand it seems that the job of determining if the content behind two different URLs are actually the same is a perfectly reasonable task for a search engine.
2 comments
[ 2.6 ms ] story [ 12.1 ms ] threadI lauched the site yesterday and had anticipated that problems might arise in the crawl process because of a naive HTML parser that cannot run JS and indeed my crawler is unable to extract anything meaningful from many sites out there, imdb.com and twitter to name a couple. I don't plan on ever running JS because I think client-side JS is a mistake all together. Must-have UI-related javascript should become HTML elements</rant>. Anyway, I'm working on fall-back parsing methods that I think will help extract text even from heavily JS-driven sites. That's coming soon.
A for me entirely new problem has arisen, though, thanks to those of you who submit urls such as these:
https://news.ycombinator.com//from?site=famouswebsite.com/us...
First of all, because of the double slashes, server returns 404. But how in the lords name am I supposed to know to discard some query params but keep others? It seems an unsolvable problem to me. Please advise.
Edit: On the one hand it seems like it's the web server's job to tell me which of these resources I should use as the canonical one. On the other hand it seems that the job of determining if the content behind two different URLs are actually the same is a perfectly reasonable task for a search engine.