i had the same reaction, but i looked to see what google's tos was for things like this and i found https://serpapi.com/ , which apparently has been around for 5 years and also offers paid services for scraping google search. if their homepage is to be believed, there are large companies using their service. i'm very surprised that google hasn't taken steps to block them.
Note that part of SerpApi's product seems to be indemnification from liability. On their pricing page, the Developer plan specifies "No Legal US Shield" and the Production plan specifies "Legal US Shield".
From their FAQ:
> What are scraper legal protections?
> SerpApi, LLC assumes scraping and parsing liabilities for both domestic and foreign companies unless your usage is otherwise illegal.
if i were evaluating serpapi, my concern wouldn't be my own legal liability. i'd be worried about serpapi getting blocked one day and my own application suffering as a result
You can use Google without agreeing to their ToS, so technically it shouldn't matter, but practically, if their lawyers come knocking, that's a different story.
I'm not sure what the connection is here. I wouldn't be so worried about violating the ToS myself if I was using the service. I'd be worried about Keyword Quill violating the ToS, getting shut down, and a service I was depending on disappearing.
SEOs and digital marketing departments needs this kind of services, especially for a huge number of this queries.
Getting this kind of data for “free” is not so straightforward when Google detects that.
That certainly helps with not having to deal with the undocumented (?) Google API yourself. The following is what chrome would use to perform autocomplete searches:
I built a little game based on a similar idea, where you enter a search query and then try and guess what the autocompletes will be. It's still very beta but feel free to try it out. Feedback welcome.
I built https://wwwww.today/ as a vanity project last year. It's in the same space but is almost completely pointless.
Pro tip: clicking through to individual results is probably the most interesting thing you can do. At some point I'll look at the aggregated data collected over the whole period. I'm curious to see which were the most fleeting autocomplete results.
25 comments
[ 3.4 ms ] story [ 61.8 ms ] threadFrom their FAQ:
> What are scraper legal protections?
> SerpApi, LLC assumes scraping and parsing liabilities for both domestic and foreign companies unless your usage is otherwise illegal.
seems easier than reverse engineering/scraping google search autocomplete
Will be ok if you just want some ideas but not if you’re tracking keywords performance in Google
curl 'https://google.com/complete/search?client=chrome&hl=en&gl=en...'
https://autocomplete-guess.blaise.gg/
I was able to get completions by just querying this endpoint:
https://www.google.com/complete/search?q=$QUERY_GOES_HERE&cl...
Pro tip: clicking through to individual results is probably the most interesting thing you can do. At some point I'll look at the aggregated data collected over the whole period. I'm curious to see which were the most fleeting autocomplete results.