Did something similar using requests, there's a rated limit around 80+ request (not sure how long it lasted though) no matter how I delay each request.
My work around is to run in on the cloud with multiple IPs, so I just switch between. Mine's a one time thing so this is workable.
If you're querying for news, Reddit might be a wise choice, they have quite a bit of free APIs afaik.
Further to the practical/technical issue of being blocked, there is a legal issue: this way of coding it (= not using an API key) violates Google's Terms and Conditions.
> is the ToS legally binding if you are just searching
Based on the LinkedIn case my guess is no, because it could be considered "generally public". But that ruling is highly suspect and likely to go to the supreme court, imo.
If it's publicly accessible, it's pretty much a public space, thus Google ToS is BS. It's the equivalent of walking by the Google office and taking a photo of the building, IMO. This wouldn't hold up in court.
Nice, until the CSV part really. I mean you have a nice DataFrame there but then it gets serialized using probably the worst format out there for totally variable data like what you scrape from a site.
Unless the file uses the actual ASCII record separator you'll end up with a CSV file which can only be read by a handful of software, after telling it explicitly what the separator and quoting rules are. And even then it's hit or miss.
And likely it does not use the RS because even though the chances it's unambiguous greatly increase and the RS was actually meant for that, software doesn't typically use it because when CSV was invented it's existence was unknown or ignored beacue it's not human readable (I guess, don't really know actually) and so the sad story began.
As you can see: I'm not a fan of CSV :) Just today I again had to waste time because at one point in the development of this otherwise fine piece of software I'm working on - even though I knew I'd regret it - I allowed it to export CSV files. Customer moved software to another machine, forgot that they once told the CSV exporter part to use the system settings, and now has CSV files with a comma separator (you know, the C in CSV). Oh the irony, that's not what they wanted.
While I couldnt agree more with your disliking of CSV, I have yet to find another more generic human-readable table file. Of all the pandas dumping methods there is no more widely available format. I find the times a CSV is utterly broken and I have to manually write the parsing logic for it quite rare actually, but then again it varies per field.
What format would you rather prefer?
parquet has a very narrow use case. it's great for internal operations but interoperability is limited unless both sides use one of the same handful of tools. csv is supported by literally every tool ever designed to work with data.
Anything unambiguous really, whatever works for the job. So unless the job is looking at very limited amount of data and drawing conclusions from it just using your brain, that means you need some kind of visalisation and processing tool. Like pandas or excel since we're talking tabular data. I.e. actually the whole premise of 'let's look at data in a plain text editor' is sort of wrong, imo. It's too limited and hence not the right tool for the job most of the times. You can't do anything but the simplest things with the data anyway in a plain text editor, so from the moment your data is more than a couple of rows in a table you're better off with another tool anyway.
CSV is totally fine, and I don't understand why people have such an issue with the format. It is very simple, here is the entire specification: the file should be encoded in UTF-8, and fields are separated by the , character. Fields may be quoted with ". If a field contains any of [",\r\n] then it must be quoted. The quote process is: replace all " characters in the field with "" and then surround the field with ". I would expect a competent programmer to be able to implement this specification as an interview question!
The existence of software that produces braindead CSVs is beside the point. Don't use that software. Just do the above and everything will work.
The answer to your first question is in your last paragraph. CSV is an abysmal format for data exchange because the standards are weak and implementations are extremely varied.
If you're writing an (en|de)coder for a specific or limited target, then sure it's fine. Otherwise I would steer clear of it.
here is the entire specification: the file should be encoded in UTF-8, and fields are separated by the , character
This exactly is part of the problem: how many people do you think would come up with this specific specification? You do realize in real life CSV gets used for anything which is plain text and separated and perhaps even quoted? Also 'should be' UTF-8? By what authority?
The existence of software that produces braindead CSVs is beside the point.
No, it is not. Especially not because there's so much software. And because it's not braindead, it's just doing something else. Which is bascially fine since there's no standard.
Just do the above and everything will work.
Yes, in your personal bubble. Beyond that it's totally insufficient.
Lots of people like to use backslash escaping, and you are so used to it that you used it in your spec even though it leaves the ambiguity of whether \n and \r are literal or the LF and CR characters. I feel like it's perfectly expressing how it "contains the seeds of its own destruction". How much data expresses a double quote as "\"" instead of """"?
It would be easy to use CSV if you never did any data interchange with it.
I don't know about being blocked. I have a vps where I used a similar naive script and it got blacklisted by google. I sometimes tunnel through it. It has been getting captchas from google the past 7 years. It is pretty permanent.
IANAL, can someone please elaborate? This sounds wrong in several ways, one of which being that Google results are almost exclusively scraped from somewhere else already.
Tangentially related, this would be a fun target for the anti-trust lawsuits. Force google to open its search results as an API service. Allow them to charge for it, sure. But make it an open API.
(Isn’t it weird they don’t offer this already? Makes you think.)
34 comments
[ 5.8 ms ] story [ 77.3 ms ] threadHave you run into issues with getting blocked by Google / issued captchas?
My work around is to run in on the cloud with multiple IPs, so I just switch between. Mine's a one time thing so this is workable.
If you're querying for news, Reddit might be a wise choice, they have quite a bit of free APIs afaik.
Based on the LinkedIn case my guess is no, because it could be considered "generally public". But that ruling is highly suspect and likely to go to the supreme court, imo.
Unless the file uses the actual ASCII record separator you'll end up with a CSV file which can only be read by a handful of software, after telling it explicitly what the separator and quoting rules are. And even then it's hit or miss. And likely it does not use the RS because even though the chances it's unambiguous greatly increase and the RS was actually meant for that, software doesn't typically use it because when CSV was invented it's existence was unknown or ignored beacue it's not human readable (I guess, don't really know actually) and so the sad story began.
As you can see: I'm not a fan of CSV :) Just today I again had to waste time because at one point in the development of this otherwise fine piece of software I'm working on - even though I knew I'd regret it - I allowed it to export CSV files. Customer moved software to another machine, forgot that they once told the CSV exporter part to use the system settings, and now has CSV files with a comma separator (you know, the C in CSV). Oh the irony, that's not what they wanted.
The existence of software that produces braindead CSVs is beside the point. Don't use that software. Just do the above and everything will work.
If you're writing an (en|de)coder for a specific or limited target, then sure it's fine. Otherwise I would steer clear of it.
This exactly is part of the problem: how many people do you think would come up with this specific specification? You do realize in real life CSV gets used for anything which is plain text and separated and perhaps even quoted? Also 'should be' UTF-8? By what authority?
The existence of software that produces braindead CSVs is beside the point.
No, it is not. Especially not because there's so much software. And because it's not braindead, it's just doing something else. Which is bascially fine since there's no standard.
Just do the above and everything will work.
Yes, in your personal bubble. Beyond that it's totally insufficient.
It would be easy to use CSV if you never did any data interchange with it.
For my thesis which required millions of datapoints, I used this tool: http://www.scrapebox.com/
IANAL, can someone please elaborate? This sounds wrong in several ways, one of which being that Google results are almost exclusively scraped from somewhere else already.
[1] https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn
(Isn’t it weird they don’t offer this already? Makes you think.)
[0] http://scraperr.com/