HN Explorer

Ask HN: Want to make $500 for scraping some data? (challenge)

6 points by gbachik ↗ HN
Open a new tab and go to google.com. Type in a band name like: "All Time Low"

You'll notice a box on the righthand side with more info about the band.

If you click the down arrow you'll see a "People also search for" section.

If you can scrape that data, in under 4 seconds(Sorry no PhantomJS), I'll pay you 500$ for your solution.

17 comments

[ 3.1 ms ] story [ 37.9 ms ] thread
stangeek
I have a working prototype. How do you want to proceed concretely?
gbachik
sorry somebody beat you to it!
dkyc
nodejs, requires node package fetch

Under four seconds, please note that this violates Google's ToS and you will likely be blocked if you do this on any significant scale.

  var bandname = "All Time Low";

  var fetchUrl = require("fetch").fetchUrl;
  var url = "https://www.google.de/search?q=";
  bandname = bandname.replace(" ", "+");

  fetchUrl(url+bandname, {
    headers: {
        'User-Agent' : 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2049.0 Safari/537.36'
    }
  }, function(error, meta, body){
    var ret = [];
    var alts = body.toString().split('alt\\x3d\\x22');
    alts.forEach(function (alt) { var cur = alt.split("\\x22")[0]; if (cur.indexOf("(")==-1)    ret.push(cur) });
    console.log(ret);
  });
gbachik
Worked like a charm! Whats your email!? I know. This is for an in house project. I don't plan on spamming google.
dkyc
In my profile!
BadCode
Just curious, I am unable to see what was the hard part in creating the solution to entail a bounty of $500?
Artemis2
Actually, OP will probably get a reCaptcha to fill after a few dozen of requests.