HN Explorer

Querying Google/a search engine through my webapp?

7 points by schtog ↗ HN
Googling "query searchengine" and the like doesnt give a good answer so i try here.

Searchengines generally do a good job of sorting out relevant data/links but they rarely ever take you all the way to the goal.

So I have some different webapps I'm working on and I would like to query Google or some other searchengine for a subject and then go through those links and present the relevant information.

I could obv write something to do this, but is it allowed? Or I have to pay for this service at Google? If so, is there some automated way to do this?

9 comments

[ 5.2 ms ] story [ 31.1 ms ] thread
FiReaNG3L
Once upon a time there was a Google API to do that, but its not available anymore. And im pretty sure scraping their result pages is against the TOS and will get you banned pretty fast.
simonw
Yahoo! have a great search API: http://developer.yahoo.com/search/

Google quietly re-introduced their search API a few months ago under the guise of the "Ajax Search API that doesn't require JavaScript" - it requires you to provide a referrer which is a bit weird, but it should work OK: http://code.google.com/apis/ajaxsearch/documentation/#fonje

schtog
Went with Yahoo's, you are right, it is very easy to use.

Just get a Yahoo-ID and download the API. The following code searches for Zlatan Ibrahimovic and prints the top 40 results:

  from yahoo.search.web import WebSearch
 
  srch = WebSearch(app_id="YahooDemo", query = "Zlatan  Ibrahimovic", results = 9)

  for res in srch.parse_results():
     print res.Url
schtog
sorry top 9 results obv.
bluelu
Google doesn't allow automated queries. An old soap api key might still work. (At least it works for the former company I worked for, but I think they also had a special deal with google)
alexstaubo
Seconding simonw's Yahoo recommendation. The API is great and the results are often as good as Google's. If you scrape Google searches, they will eventually block you.
[deleted]
(comment deleted)
kp212
I wrote a page scraper for google some time back. Its actually easier to parse their pages than yahoo's imo. You will get blocked for hours if you intend to post hundreds of times a minute though.