How does this handle sites that are rendered entirely or nearly entirely by JavaScript client side? My only real concern when looking at a new scraping solution, almost everything else can be worked with.
Thanks for your feedback. We appreciate your bringing
this to our attention, and we'll investigate. We'll follow
up with you only if we require more info or we have
additional info to share.
"Online-reviews firm Yelp Inc. alleged that Google is breaking a promise it made as part of a 2012 regulatory settlement to not scrape content from certain third-party sites including Yelp, escalating its yearslong battle against the search giant."
not according to Google, it falls under their ToS and is protected by Computer Fraud and Abuse Act.
Hey I didn't write the law, take it up with Google attorneys. Who's moving the goal post again? LOL
Also peculiar why a seemingly random HN'er would frantically defend questionable practices by OP? Are you an investor or something? a fanboy? It's very strange and raises flags. Just FYI, using alt-nicks to manipulate votes and shilling is against the rules on HN. Please be transparent about your personal stake in this matter, I have my suspicions.
edit: seems like you've hit HN's rate limiter given the sudden silence that fell amongst your nicks. My work is done here and I deserve a nice marijuana break now. Knowing I made the world a safer and fairer place today. Feels good man. Take a break from HN, get out and feel the fresh air, it's a beautiful day.
edit: all of you are ATTACKING GOOGLE as a CARTEL and thus AUTOMATICALLY EVIL. This is FALLACIOUS THINKING. Google is a NET PLUS for humanity. We get it for free. Beggars can't be choosers, especially beggars that try to exploit and profiteer without doing their own research in internet search engine.
How the fuck would you feel if I took your product, slapped my name on it and sold it? This is EXACTLY what OP is doing. There is no grey areas, its a totally BLACK and WHITE issue.
Violating trademarks and infringing on intellectual property carries jail sentencing and shit ton of fines. You are encouraging OP to go off on the far end...why? Are you relying on his service? Please clarify.
What's not so peculiar is that a 5 year old HN account with only 14 karma, and a profile that starts with
"Google has made our lives incredibly rich with knowledge and insight into our modern world. Google is the nicest monopoly out there and we should all support American corporations. I promise to defend Google from all attacks."
Is quite so insistent that web scraping a company that specialises in web scraping is inevitably such a bad thing...
It's quite obvious that at least one side of this discussion is a biased "fanboy" (as you put it)... I guess at least you're transparent about your stake in this matter in your profile...
so you are okay with somebody taking your hard work, slapping their logo and selling it without your knowledge? You are seriously going to sit there and tell me intellectual theft is suddenly not really theft.
k. im outta here.
edit: @driverdan (since you flagged all my comments to silence me) my company has been acquired and I no longer have direct control, so if you are insinuating that I am trying to squelch the competition, you are spreading libel. I'm not going to sit here and let driverdan, a self professed criminal, spread lies.
Why would you care so much as to personally write that comment? What stake do you own in serpAPI? It fringes Google's trademark and violates the Terms of Service.
If it was okay for SerpAPI to do it then everybody would be launching similar API on top of existing products. It's illegal and you are profiteering off the work of others.
I can't believe I am being downvoted for this. I am going to have to expedite the process now and reach out directly to an executive I know working at Google.
about: https://www.youtube.com/watch?v=-hL4lMcIqS4
Google has made our lives incredibly rich with knowledge and insight into our modern world. Google is the nicest monopoly out there and we should all support American corporations. I promise to defend Google from all attacks.
Google is our teacher that just gives and gives without ever asking for money, a nerdy friend that knows everything about you and your friends, it is very well a prototype of Singularity.
Without Google there would be no America. Without Google HN would not exist. Without Google Youtube and silicon valley as we know it would never have happened.
Praise be upon Larry Page and Sergey Brin, for being put on this planet to bring about Google. Praise be upon God for allowing these two smart kids to come together. Praise be upon the risk takers from Sequoia that infused money.
Praise be upon America, our free market, our freedom that reaches sea to sea.
GOD BLESS AMERICA GOD BLESS GOOGLE
In case anyone was wondering who the troll here was...
are you seriously trying to convince people that its okay to steal Google's intellectual property because they are an "evil American monopoly bent on global destruction"?
How the fuck would you feel if somebody took your lifes hard work without asking, slapped a logo and started selling it? You seriously don't have an issue with stealing other people's properties?
Who's the troll here? Someone from a former penal colony who is most likely a descendant of either a rapist, thief and a murderer from England trying to argue that intellectual property theft is not theft because Monopoly.
I'm outta here. I'll let the Google paralegals sift through this thread and before sending out subpoenas.
Reporting illegal activity is not nice? I'm sorry you got caught. Defeating captcha and bypassing IP block from Google servers would fall under CFAA, which they nailed 3taps for.
Yeah, we know you are not a lawyer and neither am I, but writing I ANAL, doesn't really exempt you from the law...im sure it doesn't take a lawyer to realize this.
see 3taps vs Craigslist and the CFAA ruling. Public data is fine, its just when you break their captcha and bypass their IP ban, it becomes an issue. Which you've pretty much admitted in your reddit thread.
You can try defending yourself in court against the world's best legal team. Any capitulation now is better than losing your entire life savings over in the subsequent legal defense fees alone.
edit: whats the matter? run out of alt-nicks? LMAO. dude you make it way too obvious. inactive accounts suddenly converging on this particular thread using the same argument over and over to censor my comments...not even bothering to change writing styles...
IANAL and this does not constitute legal advice. Craigslist v. 3Taps has been overturned. There are more recent rulings.
On another level, EFF fights to ensure computer access to public data stays legal. It's a worthy fight.
We have no relationships with the other commenters. Anyway, I won't reply to this thread anymore. You can reach me via email, julien _at_ serpapi.com, if you want to continue this discussion.
> 3Taps and PadMapper were companies that partnered to provide an alternative user interface for browsing Craigslist's housing ads. In doing so, they scraped Craigslist's site for data, which Craigslist did not approve of. Craigslist sent both companies a cease-and-desist letter and blocked their IP addresses, but this did not stop 3Taps from scraping through other IP addresses. Craigslist then sued, resulting in this case.
I think you may be in the deep-end now. Good luck with the sharks.
Your audacity is admirable but its foolhardy. You based your entire business on incomplete legal ruling without consulting a lawyer I presume.
How the fuck would you feel if somebody took your life's work, slapped their logo and started selling it? I'm sure you can see the ethnical issues of this and the legal ramifications.
no. it took less than 5 minutes. I'd gladly give it up if it means catching the bad guys that violate ToS!
did you just login after weeks of inactivity just to defend OP, like the other nicks on this thread ;)
Perhaps you should be speaking to your lawyer, instead of wasting time trying to censor me with obvious alt nicks.
I'm the least of your problems at this point, as I've gotten a confirmation email from Google's Legal team. Yeah dude, this would piss off any monopoly.
Lol, nah been logged in for sometime, came across your wonderful comment, and found it funny that the internet white knight stereotype exists at this extreme. In this case, someone willingly narcing on a mega corporation's behalf, for no compensation, over little damage to said mega corporation monetarily/competition-wise, and only to serve what they see as being just and proportional consequences (in reality overblown) to a fellow HN member
How the fuck would you feel if somebody took your original hard work, slapped their logo, and started peddling it to unsuspecting customers? You are seriously going to ignore this key legal and ethics issue? Give me a fucking break and do your god damn research first on the subject before you run your mouth.
I don't think you know what narcing is. It comes from employees of drug cartels that turn against them and testify against their illegal activities. Is google a dangerous drug cartel? I guess they are a data cartel of sorts. Not sure why that would be used to argue that they are evil....yet you are still using Google to do research....hypocrisy much?
If we want to keep Google free, we need to help them.
Google lawyers might not see it as "exaggerated" more of a "infringement of property and trademark" or even pushing for "computer fraud and abuse act".
Don't hate me, hate Google. More likely they have been monitoring serpapi already and I'm just another uncompensated sucker that reported it.
I'll let you know as soon as I hear from Google's legal team, since you seem to be mysteriously fixated on defending serpAPI....like literally right after OP has posted a reply...timing is highly suspect
how would you feel if somebody took your product you spent years on, slapped their own logo and started selling it? Intellectual theft is theft regardless....and here you are justifying it. WOW.
are you seriously defending OP's operation? Even when its questionable at best ethically? Google doesn't want people using their search results to make money off of it. If they did, they would release it like Bing API. I'm sure you can see why this would be worthy of Google's attention and why enforcement of such rules is necessary. Otherwise, spamming marketers would easily exploit something like serp api, you are not foolin anybody.
In any case, I saved OP from an even bigger financial ruin. He might just get off with a cease & desist letter. It depends on how much revenue he generated using Google's product without licensing. It's clear OP found out about 3taps vs craigslist, a heavily debated ruling on HN, just today.
Just because Google is a monopoly, doesn't justify civil disobedience here, which seems to be what some are arguing for.
Apparently he no longer owns scrape.it. He previously replied to my comment to say he sold it and to threaten legal action. He deleted the comment and sent me the following email:
> If this comment is not removed immediately I am going to have to involve legal. Scrape.it is no longer under my direct control, it has been acquired as of Feb 2019.
> Cheers,
> John Kim
> Scrape.it Founder
So I guess he was the founder but is no longer involved.
It's not doxxing. I found his name through his past HN posts prior to his threatening email. Anyone can figure out who he is with less than 10 min of searching.
yeah well im sure anyone can look up your criminal records, or was it wiped as a part of your plea deal? Do you even feel empathy for the people you fucked over in the past? As somebody who was a victim of a financial crime, it is the single most worst thing you can do to a family, as Im sure it was for your parents. When my parents were victimized by ID thiefs like you you traumatized an innocent kid just like your current offsprings. Thiefs like you broke my family apart and drove my mother insane. She's just not the same since, and it happened around the same time you were committing crime. Fuck you, you fucking sociopath. Does your kids know their daddy ruined other families lives?
It saddens me to see your past glorified in mainstream articles and you milking it very well.
What skin do you have in SerpAPI exactly, you don't exactly strike me as an angel here.
Which is more likely here, I'll take a jab: a criminal thief, that sees some romantic notion of mob justice for "narcing"? Or perhaps you were the beneficiary of SerpAPI (hey back when I was a freelancer this was written in Visual Basic), and now jaded that its been swept under your feet, which explains the sudden influx of downvotes.
It puzzles me as to why someone would be willing to risk their current breadwinner that probably puts food on your kids table.... for a complete stranger from a thread you just "stumbled upon" like the rest. Totally no bandwagoning on HN folks /s
41 comments
[ 2.8 ms ] story [ 150 ms ] threade.g.:
- https://github.com/serpapi/test-knowledge-graph-desktop/tree...
- https://travis-ci.org/serpapi/test-knowledge-graph-desktop
- https://github.com/serpapi/test-organic-results-desktop/tree...
- https://travis-ci.org/serpapi/test-organic-results-desktop
- https://github.com/serpapi/test-news-results-desktop/blob/ma...
- https://travis-ci.org/serpapi/test-news-results-desktop
Producing reliable scrapers and parsers is very hard. Testing as much possible is the only way to go. Smart use also of JSON schema on Spidermon.
edit: dude, you need to stop, or you are going to get sued, I'm serious.
https://www.reddit.com/r/webdev/comments/8a2zx9/weve_just_la...
I've alerted legal@google.com about SerpAPI
It also ignores the fact that Google has scraped many sites to build its own profitable products and features over the years.
https://www.wsj.com/articles/google-rival-yelp-claims-search...
https://www.eff.org/deeplinks/2015/06/padmapper-and-3taps-se...
Hey I didn't write the law, take it up with Google attorneys. Who's moving the goal post again? LOL
Also peculiar why a seemingly random HN'er would frantically defend questionable practices by OP? Are you an investor or something? a fanboy? It's very strange and raises flags. Just FYI, using alt-nicks to manipulate votes and shilling is against the rules on HN. Please be transparent about your personal stake in this matter, I have my suspicions.
edit: seems like you've hit HN's rate limiter given the sudden silence that fell amongst your nicks. My work is done here and I deserve a nice marijuana break now. Knowing I made the world a safer and fairer place today. Feels good man. Take a break from HN, get out and feel the fresh air, it's a beautiful day.
edit: all of you are ATTACKING GOOGLE as a CARTEL and thus AUTOMATICALLY EVIL. This is FALLACIOUS THINKING. Google is a NET PLUS for humanity. We get it for free. Beggars can't be choosers, especially beggars that try to exploit and profiteer without doing their own research in internet search engine.
How the fuck would you feel if I took your product, slapped my name on it and sold it? This is EXACTLY what OP is doing. There is no grey areas, its a totally BLACK and WHITE issue.
Violating trademarks and infringing on intellectual property carries jail sentencing and shit ton of fines. You are encouraging OP to go off on the far end...why? Are you relying on his service? Please clarify.
"Google has made our lives incredibly rich with knowledge and insight into our modern world. Google is the nicest monopoly out there and we should all support American corporations. I promise to defend Google from all attacks."
Is quite so insistent that web scraping a company that specialises in web scraping is inevitably such a bad thing...
It's quite obvious that at least one side of this discussion is a biased "fanboy" (as you put it)... I guess at least you're transparent about your stake in this matter in your profile...
k. im outta here.
edit: @driverdan (since you flagged all my comments to silence me) my company has been acquired and I no longer have direct control, so if you are insinuating that I am trying to squelch the competition, you are spreading libel. I'm not going to sit here and let driverdan, a self professed criminal, spread lies.
If it was okay for SerpAPI to do it then everybody would be launching similar API on top of existing products. It's illegal and you are profiteering off the work of others.
I can't believe I am being downvoted for this. I am going to have to expedite the process now and reach out directly to an executive I know working at Google.
HOW THE FUCK WOULD YOU FEEL IF SOMEBODY TOOK YOUR HARD WORK, SLAPPED A LOGO, AND STARTED SELLING IT? YOU ARE OKAY WITH THAT???!
I'm taking a snapshot of this entire thread, just in case.
http://archive.is/eDBM6
In any case, it won't help you against Google's army of Ivy league attorney.
about: https://www.youtube.com/watch?v=-hL4lMcIqS4 Google has made our lives incredibly rich with knowledge and insight into our modern world. Google is the nicest monopoly out there and we should all support American corporations. I promise to defend Google from all attacks.
Google is our teacher that just gives and gives without ever asking for money, a nerdy friend that knows everything about you and your friends, it is very well a prototype of Singularity.
Without Google there would be no America. Without Google HN would not exist. Without Google Youtube and silicon valley as we know it would never have happened.
Praise be upon Larry Page and Sergey Brin, for being put on this planet to bring about Google. Praise be upon God for allowing these two smart kids to come together. Praise be upon the risk takers from Sequoia that infused money.
Praise be upon America, our free market, our freedom that reaches sea to sea.
GOD BLESS AMERICA GOD BLESS GOOGLE
In case anyone was wondering who the troll here was...
How the fuck would you feel if somebody took your lifes hard work without asking, slapped a logo and started selling it? You seriously don't have an issue with stealing other people's properties?
Who's the troll here? Someone from a former penal colony who is most likely a descendant of either a rapist, thief and a murderer from England trying to argue that intellectual property theft is not theft because Monopoly.
I'm outta here. I'll let the Google paralegals sift through this thread and before sending out subpoenas.
IANAL and this does not constitute legal advice. Scraping public data is legal in the US. See the couple of cases which did go to court.
Yeah, we know you are not a lawyer and neither am I, but writing I ANAL, doesn't really exempt you from the law...im sure it doesn't take a lawyer to realize this.
see 3taps vs Craigslist and the CFAA ruling. Public data is fine, its just when you break their captcha and bypass their IP ban, it becomes an issue. Which you've pretty much admitted in your reddit thread.
You can try defending yourself in court against the world's best legal team. Any capitulation now is better than losing your entire life savings over in the subsequent legal defense fees alone.
edit: whats the matter? run out of alt-nicks? LMAO. dude you make it way too obvious. inactive accounts suddenly converging on this particular thread using the same argument over and over to censor my comments...not even bothering to change writing styles...
On another level, EFF fights to ensure computer access to public data stays legal. It's a worthy fight.
We have no relationships with the other commenters. Anyway, I won't reply to this thread anymore. You can reach me via email, julien _at_ serpapi.com, if you want to continue this discussion.
https://en.wikipedia.org/wiki/Craigslist_Inc._v._3Taps_Inc.#...
> 3Taps and PadMapper were companies that partnered to provide an alternative user interface for browsing Craigslist's housing ads. In doing so, they scraped Craigslist's site for data, which Craigslist did not approve of. Craigslist sent both companies a cease-and-desist letter and blocked their IP addresses, but this did not stop 3Taps from scraping through other IP addresses. Craigslist then sued, resulting in this case.
I think you may be in the deep-end now. Good luck with the sharks.
Your audacity is admirable but its foolhardy. You based your entire business on incomplete legal ruling without consulting a lawyer I presume.
How the fuck would you feel if somebody took your life's work, slapped their logo and started selling it? I'm sure you can see the ethnical issues of this and the legal ramifications.
did you just login after weeks of inactivity just to defend OP, like the other nicks on this thread ;)
Perhaps you should be speaking to your lawyer, instead of wasting time trying to censor me with obvious alt nicks.
I'm the least of your problems at this point, as I've gotten a confirmation email from Google's Legal team. Yeah dude, this would piss off any monopoly.
Your most welcome.
How the fuck would you feel if somebody took your original hard work, slapped their logo, and started peddling it to unsuspecting customers? You are seriously going to ignore this key legal and ethics issue? Give me a fucking break and do your god damn research first on the subject before you run your mouth.
I don't think you know what narcing is. It comes from employees of drug cartels that turn against them and testify against their illegal activities. Is google a dangerous drug cartel? I guess they are a data cartel of sorts. Not sure why that would be used to argue that they are evil....yet you are still using Google to do research....hypocrisy much?
If we want to keep Google free, we need to help them.
Google lawyers might not see it as "exaggerated" more of a "infringement of property and trademark" or even pushing for "computer fraud and abuse act".
Don't hate me, hate Google. More likely they have been monitoring serpapi already and I'm just another uncompensated sucker that reported it.
I'll let you know as soon as I hear from Google's legal team, since you seem to be mysteriously fixated on defending serpAPI....like literally right after OP has posted a reply...timing is highly suspect
Wow, what a hero!
are you seriously defending OP's operation? Even when its questionable at best ethically? Google doesn't want people using their search results to make money off of it. If they did, they would release it like Bing API. I'm sure you can see why this would be worthy of Google's attention and why enforcement of such rules is necessary. Otherwise, spamming marketers would easily exploit something like serp api, you are not foolin anybody.
In any case, I saved OP from an even bigger financial ruin. He might just get off with a cease & desist letter. It depends on how much revenue he generated using Google's product without licensing. It's clear OP found out about 3taps vs craigslist, a heavily debated ruling on HN, just today.
Just because Google is a monopoly, doesn't justify civil disobedience here, which seems to be what some are arguing for.
> If this comment is not removed immediately I am going to have to involve legal. Scrape.it is no longer under my direct control, it has been acquired as of Feb 2019.
> Cheers,
> John Kim
> Scrape.it Founder
So I guess he was the founder but is no longer involved.
https://news.ycombinator.com/newsguidelines.html
in your profile you wrote:
> cryptocurrency. Former blackhat / ID thief.
oh. I see what we are dealing with here. a criminal.
It saddens me to see your past glorified in mainstream articles and you milking it very well.
What skin do you have in SerpAPI exactly, you don't exactly strike me as an angel here.
Which is more likely here, I'll take a jab: a criminal thief, that sees some romantic notion of mob justice for "narcing"? Or perhaps you were the beneficiary of SerpAPI (hey back when I was a freelancer this was written in Visual Basic), and now jaded that its been swept under your feet, which explains the sudden influx of downvotes.
It puzzles me as to why someone would be willing to risk their current breadwinner that probably puts food on your kids table.... for a complete stranger from a thread you just "stumbled upon" like the rest. Totally no bandwagoning on HN folks /s