Show HN: Headless Chrome Crawler (github.com)

`page.$(query)` or `page.$$(queryAll)` to be more specific

https://github.com/GoogleChrome/puppeteer/blob/v1.1.0/docs/a...

yes I've used puppeteer, but I couldn't see where it exposed the page object, but looking closer I saw

HCCrawler.launch({ // Function to be evaluated in browsers evaluatePage: (() => ({ title: $('title').text(), })), // Function to be called with evaluated results from browsers onSuccess: (result => { console.log(result); }), })

which doesn't look that good for my other worry, that you come to a dynamic JS built up page, load the DOM and evaluate in some milliseconds and then that DOM is changed after you've done your incorrect evaluation.

most naive code I see is like the following

    const page = await crawler.browser.newPage();
    await page.goto(url);
    await page.waitForSelector("a[href]");
    const hrefs = await page.evaluate(
    () => Array.from(document.body.querySelectorAll('a[href]'), ({ href }) => href)
    );

and then you do something with hrefs.

However if you have a page that loads with 4 links defined, does its script and then ends up 100+ links you miss the 100+ links. I notice people often failing to account for this in their crawlers, so I wondered if this one was.

I've done this recently actually. Take a look at the yukinying/chrome-headless-browser[0] image. You'll need to run with the SYS_ADMIN capability and up the shm_size to 1024M (you can workaround the SYS_ADMIN cap with a seccomp file but I didn't have much luck with that). Other than that oddness it works pretty well (and with Puppeteer 1.0, with far fewer crashes).

[0]: https://github.com/yukinying/chrome-headless-browser-docker

I recently did this in a Docker.

Let me quickly add instructions here, first you need to install some dependancies, add the following to dockerfile:

  RUN apt-get install -y gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget

Secondly, launch puppeteer with --no-sandbox option:

  browser = await puppeteer.launch({
      args: ['--no-sandbox'] /*, headless: false*/
    })

That should do it.

I can't puppeteer to screen capture websites very well, crashes entirely after a few

Unless something has changed that I missed, you can install extensions (I complained when the default args messed this up [0]). For example, I built something that uses puppeteer and an extension to capture audio and video of a tab [1]. It's just headless mode that doesn't allow extensions [2] (which I now realize is probably what you meant).

0 - https://github.com/GoogleChrome/puppeteer/issues/850 1 - https://github.com/cretz/chrome-screen-rec-poc/tree/master/a... 2 - https://bugs.chromium.org/p/chromium/issues/detail?id=706008

Thanks. Wish it was simpler. It seems overkill to have to an extra proxy in the middle with no auth to authenticate with the one with auth, just to make headless chrome working.

Awesome. I need to figure out a way to make it work with our Ruby code, but it shouldn't be that hard. Thanks.