Ironic because LinkedIn scrapes and collects every scrap of contact info they can find. I've got people I sent an email to once in 199? suddenly popping up suggesting we know each other.
"HiQ scrapes data about thousands of employees from public LinkedIn profiles, then packages the data for sale to employers worried about their employees quitting".
A few weeks ago I was looking up public profiles on LinkedIn and I noticed what I interpreted to be some network-side fingerprinting of some kind. The first couple profiles came up, but from that point I was only served a sign up page. It didn't matter if I changed browsers or spawned new incognito sessions.
That crossed my mind, but it'd mean they'd be willing to lock out large networks behind a NAT. I was wondering if there's something more sophisticated available.
Also, they are just locking out users that aren't signed in. If a user has a login, they are probably much more likely to err on the side of caution before locking down access in some way.
>One plausible reading of the law—the one LinkedIn is advocating—is that once a website operator asks you to stop accessing its site, you commit a crime if you don't comply.
I'm fine with that.
But I'm also fine if they do the like the article suggests and require anything non-scrapable to be behind an account prompt, even if everyone with a account can access it.
I don't think it's fair to make Linked In foot the bill for someone else's business. They shouldn't have to serve that content to people who aren't actually their users.
This is the crux of the issue, it's my impression that the CFAA doesn't have any language that sites have to take precautions to prevent unauthorized access.
If a Google-like competitor started, all Google would have to do to crush them is demand big-name sites formally prohibit the competitor from accessing their content or risk being delisted. And magically, it becomes impossible/illegal to build a duck-duck-go.
Isn't that monopolistic behaviour? I suspect that it might violate some laws, though I don't know anything in particular. On the other hand, why couldn't DDG just do what Google does and say "robots.txt prevents us from getting a description for this site"?
On one hand I'm against the cartel beahviour of Google doing something like that, but on the other hand, if Google asks and the other company agrees to block DDG, why shouldn't that be allowed?
> They shouldn't have to serve that content to people who aren't actually their users.
And insurance companies only want to serve people who won't claim etc. LI is free to force users to usage agreements, but why should third-parties (taxpayers) foot the bill for enforcing them?
Hard to say...they were charged with both wire fraud and "hacking" (CFAA).
The EFF said this, so they seem to agree that it set a bad precedent, even for those just scraping.
"Under the government's theory, anyone who disregards – or doesn't read – the terms of service on any website could face computer crime charges," said EFF civil liberties director Jennifer Granick in a press release at the time. "Price-comparison services, social network aggregators and users who skim a few years off their ages could all be criminals if the government prevails."
They were being sentenced to prison but not for scraping, their business were a fraudulent economic activity. They broke the terms of buying those tickets by impersonating legitimate customers. I don't see the connection with scraping.
> If you have permission to view the page, then you have permission to scrape the page.
This is false. Web pages expect scrapers to respect robots.txt and users to respect the terms and conditions. From what I understand, the CFAA law was not meant for these violations, and the fight is whether or not this law can be used to win the trial. However, other laws can be put into place to address these situations.
Providing the information on a publicly accessible page provides permission. It's like posting a letter on a public noticeboard and demanding that certain people not read it.
not so much "demanding that certain people not read it" but demanding that certain people not take certain action with the information on the notice board.
There is a distinction between a person reading one page and a scraper reading all pages.
First, the scraper consumes a lot more bandwidth (which some websites might have real trouble with). Second, the aggregated scraped data is a lot more valuable than what any single person receives from reading a few pages.
Websites could (and sometimes do) limit the number of views and the frequency of accesses from the same IP, which allows users to read the page and discourages scrapers without "taking it down".
"this scraping violated the Computer Fraud and Abuse Act, the controversial 1986 law that makes computer hacking a crime. HiQ sued, asking courts to rule that its activities did not, in fact, violate the CFAA."
I'm following this story with a lot of interest. I've done (and still do!) a lot of data crawling/scraping. In the past I've worked on so-called "alternative data" collection and analysis for financial forecasting.
Without going into too much detail, a lot of hedge funds have teams constantly searching for kernels of data that can contribute some kind of signal for market movements. This data can come in the form of satellite imagery for oil tankers or manufacturing centers, but it can also come from the very creative use of scraped and aggregated data. It's typically very difficult to identify, collect and analyze on a technical level (as 'chollida1 has lamented in the past: normalization, labeling/bucketing and analysis of disparate data across different formats, sources and processing timeframes is a pernicious problem at this scale). From a compliance standpoint there are also generally strict requirements governing legality of use.
Depending on the specific data, you might be capable of predicting earnings or broader market movements with a <5% margin of error each quarter for years at a time (I've personally seen and worked on projects with <1%, but that's the exception, not the norm). That tactic is usually found at discretionary funds; at quantitative funds the uses are much more abstract and cross-pollinated so as not to target single-equities, but rather holistic trends. Regardless, every fund is using data in some way these days; it's just a matter of how sophisticated, creative abstract they get in their analysis of it.
hiQ Labs doesn't collect data for this specific purpose, but it is absolutely related. In the past I have stayed away from crawling LinkedIn and Yelp precisely because they are very litigious (regardless of the eventual outcome and legality). Now that there's another relatively high profile case out in the open like this, I'm interested in seeing how it proceeds and what the ramifications will be for companies that collect data across a wide range of uses. As Grimmelman mentioned in the article, this can impact a lot of types of businesses, not just those in the same space as hiQ. Outside of finance I am familiar with many tech companies which (openly or otherwise), kickstarted what are now widely known enterprises through cleverly crawling or scraping massive amounts of data.
"satellite imagery for oil tankers" - Interesting.
If hedge funds floating drones above oil tankers, I'd guess they'd be accussed of corporate espionage / spying / invasion of privacy?
Ok, so oil tankers are big and "in the clear". What if $TANKERCORP floats big parachute balloons above its tankers to imply "looking past these is unauthorized viewing"?
Then if a HedgeFund gets a clever angle on a satellite photo.. is that the equivalent of breaking a lock, or violating CFAA?
Satellite imagery like this is legal to within a few feet of practical resolution, pretty much anywhere. The effective countermeasure is hiding things from a satellite, not attempting to sue satellite operators for flying overhead. I'm not aware of a specific law against using anything that is literally viewable from the sky, at least in the United States (someone can correct me if I'm wrong, but last I checked Google Maps blurs out some locations or keeps them outdated because the government requests it, not because of a formal law forcing them to do so).
There are two other notes in response to your question:
1. Drones are different from satellites, and are more susceptible to regulation in the way you're positing because they can be prevented from flying above specific areas. However, most of the same problems with countering them apply, because drones can record better three dimensional footage. In your specific example, if a tanker disguised itself overhead, it would still be legal to have a drone monitor the tanker from the sides, as long as doing so didn't break any law set by the FAA.
2. Drones are actively used these days for things like monitoring production facilities ("how many cars come out of this factory" for an oversimplified version). If they have to monitor from a distance, so be it, they'll do it. The effective countermeasure here is to have a huge amount of land that can't provide any intelligence, because the drones aren't allowed to fly over it and can't see far enough in to the facility.
There's definitely a productive ethics discussion that can be had here, but the legal precedents don't really allow for combatting these techniques right now. If it's public, it can be collected, ingested and used in an algorithm to determine alpha.
> someone can correct me if I'm wrong, but last I checked Google Maps blurs out some locations or keeps them outdated because the government requests it, not because of a formal law forcing them to do so
In the Eastern European country from where I'm from (a NATO member) the Google StreetView car even got to photograph and publicly put on the Internet the outside of military and air bases with clear signs of "do not take photographs" visible on StreetView itself. It's funny, my company also used to work in this space (local business directory with business addresses, photos of said business etc) and one of my former colleagues got detained for a day because of taking photos of businesses in the downtown area of one of the biggest cities in my country. He hadn't seen that there was a military "objective" in his line of view (probably some military HQ or someth, not a proper military base with tanks and trucks). Talk about the advantages of being an internet giant like Google..
Hedge fund researchers have also chartered private airplanes to fly over oil storage facilities and use infrared cameras to check tank levels. In the USA at least this is completely legal as long as they observe regular FAA flight rules. For the oil market as a whole this is a good thing since it helps price discovery.
It's not practical or safe to put a huge parachute or balloon over a tanker to block overhead imagery. Any sailor can tell you it just wouldn't work.
> Outside of finance I am familiar with many tech companies which (openly or otherwise), kickstarted what are now widely known enterprises through cleverly crawling or scraping massive amounts of data.
Google comes to my mind. I'm only partly joking, as it seems to me that the line between search engine web crawling and other forms of web scraping is very thin.
It's more than thin: just scrape a few more sites, make a public search interface and say you are working on a search engine. In the meantime, you might analyze the data for other (not so) related purposes.
> Inside, are certain sites more worthwhile - and which ones (eg reddit, eBay, trade union websites, whatever)
Yes, absolutely. For many purposes websites that sell their own data are less useful (less signal exclusivity). Specific sources of data will be much more valuable depending on what the data is about.
> How about scraper brokers? Do they exist?
Yes. You're not getting access without an NDA in addition to paying quite a lot.
> Are there scammy scrapers? Make up BS and sell as scraped data?
That depends on how easy it is to verify the data. For most of what you'd term "alternative data" you'll know if it's real in 2 - 12 weeks, and it's not sustainable to sell crap.
But a lot of parties scrape dodgy financial timeseries data (ticks and quotes on equities or options) and sell it, priced as though it were tick data when it's barely accurately OHLC. They mostly sell this sort of data to amateurs who don't realize tick data is expensive for a reason.
> How big is this?
Very big. Most hedge funds ingest a lot of data whether they curate it internally or source it from elsewhere.
Well, it's pretty simple to legally dodge these kinds of threats.
If you scrape regularly, then pick up a dozen or more machines around the world, in less than friendly areas to US law. Pay with a rechargeable credit card or bitcoin. And the buy servers and set up a hadoop cluster that handles scan-jobs.
The worst case scenario is that LinkedIN, YELP, and others get some of your servers shut down. Wash, rinse, repeat.
EDIT: please note, this was only a thought-game to bypass rude and destructive laws like the CFAA, which weaponizes TOS'ses, EULA's, and other implicit contracts of adhesion (as in, you have to agree to see). Ideally, we would be better off with both these laws to have a sane scope, and for companies to not expect things to happen with content in public.
You're too technical, feels like.
Usually it's done in even simpler way - you have 3rd party provider from one of this countries (or who aggregates data from you), who does all scrapping and data cleaning for you for reasonable price.
Alternate data doesn't even need to be as sexy as satellite photos, hell you almost certainly want the data that isn't sexy, the stuff people haven't thought of because it's too boring. Alternative data vendors above all want sales, and even the funds themselves want things to show off to clients. This gives you great opportunities to look at the alternative data they aren't touching.
Given this is a predominantly a web development community it always surprises me how little creativity there is in the articles on investing. Neural networks and machine learning sound cool but the reality is almost none of the readership would be able to make any money off them.
Simply tracking how many sales or users exist in databases by watching sequential IDs should be the go-to method for any web developer trying to get an edge. I would have expected HN to have articles where people are getting creative on that, ie trying to use measures of entropy on usernames to get rough subscriber numbers etc.
Even plain scraping of prices etc, is often full of great insight that is ignored. If a grocery store drops their prices in profitable categories against their competitors, that could be the signal about an incoming price war for an entire sector. There's a lot more information in that than social media feeds and all the other sorts of sexy data that get coverage in the media.
The reason hedgefunds look at satellite images and oil tankers is because everyone looks at sequential ids and price changes so that doesn't give an edge.
That's simply not true - equity analysts can cover anywhere between 5 and 500 stocks, do you really think they have the time or skill set to track all of that? It really is laborious, grueling work.
If you look at the possible returns the equity market is going to make from a stock in a dollar value, and how much research spend is as a percentage of that, you'll quickly see it doesn't pay for much.
You can tell simply by looking at the broker research - that's probably the extent that analysts take things.
The big stocks obviously have a lot of it happening. (eBay listings, airline pricing etc is obviously touted a lot)
But once you start to go down to the mid caps, you enter a void where there isn't much heavy data focused research done, and it's very possible you can have a better gauge of the business than any other investor on the planet once you pull out this data out.
That's a pretty literalist, one-size fits all approach to policy. I don't think it's a good framework to use for applying ethics considerations.
If I can walk near a pool, should I also be able to run? Is running anything more than faster walking? If I'm allowed to be around the pool walking with my entry ID, should I also be allowed to place my ID on a little motorized car and make it dart around the pool really fast? Should I be able to duplicate my badge, put it on a bunch of little cars and direct them to quickly get all the floaties before anyone else? How about giving them all their own fake IDs? Now all the same questions, except there is a sign that explicitly prohibits all of these examples except for walking.
It seems disingenuous to argue that the automation and rapid increase of a thing should be allowed just because a thing is allowed. That doesn't typically match our intuitive notions of ethics in other parts of society, like driving or walking around a pool. Yes, you can walk around a pool as much as you want, but if you change to running then you have fundamentally altered your behavior through increased capability, not merely done "more of walking" to utilize more of your freedom.
I suppose a natural counterargument to this analogy might be that running around pools is unsafe, and scraping is not unsafe in the same way. But my point here is establishing that a behavior intrinsically changes into a different behavior if you increase the speed at which you're doing it or the capability at which you can do it.
I'm interested in this point. I was under the impression that the CFAA would apply here because they've explicitly stated what users are authorized and not authorized to do on their platform. Other than incur shame from the tech community they don't have to actually prevent the unauthorized access or use.
Ostensibly "ethical considerations" are a precursor to a law being ratified, and will also be used when a law is not sufficiently specified as to obviate the decision. If a party is not denying their activity in court, but rather claiming that it doesn't match existing laws, this would come into play.
But if you contest that or have an issue with the specific terminology I used, perhaps you'd prefer this terminology instead: "I don't think this is a good framework for interpreting the CFAA and establishing legal precedent."
I like this analogy. I think maybe the answer would be to include technical limitations, like API limits so scrapers can only work as fast as a human. Or maybe captcha interaction like select all images of a car could be integrated? I think it should be up to the pool operator to make sure people follow the rules they put in place, not call the cops because someone is running.
The pool has the right to kick you out, same as any website. The pool cannot call the police and charge you with a felony for misusing their resources.
Does the pool have any recourse if you proceed to bypass the ban? Do you have to re-enter to pool to bypass it, or does sending in confederates with their own badges to continue your work also bypassing the ban? How about sending in new motorized cars?
The analogy is starting to break down, but I think it's still instructive for the problem of applying a simple first principles approach.
There is a legal concept known as "attractive nuisance"[1]. If I have a pool and neighborhood kids come to play and someone gets hurt, it's my fault. Even if I was away from my house and never gave permission (or explicitly forbade them from swimming), if I don't have proper access controls in place, the courts say it is too tempting for the neighbors to just come over and swim. I need to put up a locking gate to keep them out.
Likewise in some high-crime jurisdictions, if you did not lock your car you are liable for it getting stolen or broken into[2]. An unlocked car is too tempting for some people to just walk past and not take it.
I know it might sound crazy but you could make the argument that a massive pool of highly-structured and very valuable data just sitting out in the open is an attractive nuisance and steps should be taken to put it behind a locked gate. Once that requirement has been satisfied, normal trespassing laws apply.
Laws like that are ridiculous. You can see that by looking at how the reasoning does not expand to certain areas.
For example, if a woman walks down a dark alley wearing short skirts and gets raped, it isn't her fault. I mean can you imagine if we said "well, she was just an attractive nuisance!" The judge would throw the book at you.
I don't think it's true, and the source doesn't back it up. Even if it's illegal to leave your car unlocked, that doesn't mean it's your fault if someone steals it. Presumably you'd both be liable.
The website can't really kick you out though, it can only kick your agent out and you can trivially create a thousand more. The website can politely ask you to stop just like the pool, but it can't actually do anything if you ignore it.
Except block your IP address, or your user agent, or the pattern your software makes when it connects.
Yes, that will cause potential issues for other people, which is why they tend not to do that, but if you trivially create a thousand more agents, and potentially trigger a degradation of service, how are you different to the people who block junctions at traffic lights?
I'm not keen on inconveniencing people, and "it's not that bad" is a poor argument for doing something that someone has explicitly asked you not to do.
I think you agree with me. If someone is abusing your website, or even just doing something you've asked them not to do, you can't literally kick them out like you could in person. There's all sorts of techniques for blocking whatever they're doing, but if they're determined and your website is still accessible, they can come back.
That's why it might be reasonable for laws around this sort of thing to be different in the virtual world.
> The pool cannot call the police and charge you with a felony for misusing their resources.
Once they have withdrawn permission, they can call the police and you can be charged with trespass, though that's usually a misdemeanor rather than a felony.
The way HTTP works is there is a request and response. If LinkedIn does not want to provide a response, they don't have to. If they don't want to accept requests, then maybe the World Wide Web isn't for them.
The publisher of the information is ultimately responsible for the disclosure of the information. How it is read is of no consequence, as the information has been provided to be read. Certainly, there are issues about resource usage i.e. heavy readers, such as scrapers, but throttling is a perfectly acceptable approach to overuse of resources for both parties, as access is still available over managed resources.
The nature of the original complaint is authorised access to publicly published information. Again, if you do not want people to read publicly published information, do not publish it publicly.
And as a side note - we don't need inappropriate analogies; the web is real, we can discuss the real issue.
What LinkedIn is doing seems more like a pool threatening to sue people that look at the facility itself, from outside of their property. It's not clear whether viewing a public website is more (reasonably) similar to entering someone's property versus looking at their property from outside of it. To me, it seems much more like looking at it from outside. But, fortunately for LinkedIn, they can prevent people from viewing their site! (They just have to figure out for whom they want to do so.)
At the risk of bringing out the "you wouldn't download a handbag" brigade - you could make the argument that just because I leave my bike unlocked on the street doesn't give you a right to ride it off.
Even if you go for "forgiveness rather than permission", if you ride it a second time after I've told you I don't want you riding it, then you're in the wrong.
Obviously, there are philosophical arguments to be made about the status of information online, but if the information were provided to LinkedIn, to be used on LinkedIn properties, I as a user would take umbrage with it being taken by a 3rd party, even if it were viewable publicly.
The bike example is a very poor analogy - the data isn't removed from LinkedIn, it's merely copied. It doesn't matter how many times the site is scraped, the data is unchanged and still available. A better analogy would be me taking a photo of the bike while walking past. It shouldn't matter how many times you tell me to stop, if it's on public property you can't really stop me from taking the photo.
If you have an issue with that, you should be moving your bike somewhere people can't take a photo from the public street. Not have someone creatively interpret a law that says where I am is suddenly not public property, because you asked me to stop using my camera.
Ok, if I left the prototype for a new bike on the pavement, and you came along and 3D printed an exact replica. Sure, the original still exists, but you just violated trademark/registered design laws because it was there. It's not the same as a photo because a photo of a bike doesn't give you the same value as the actual bike, whereas scraping the content of a webpage does give you the same value.
Perhaps my analogy wasn't great, but the grey area is around going to LinkedIn's server (whether or not this is "public" or their property they allow you access to is another philosophical question, though in the eyes of the law it appears it's the latter), deliberately extracting value from it, and then getting annoyed when you're asked not to.
Inherently it seems as though it's the old question of whether a server is public, or private but accessible (like those POPs [0] there was a thread on recently).
Not sure why you're being down-voted, the Web were designed to be public. If you want to prevent me from taking pictures of the exterior to your Cafeteria on the public street you'd have to build a wall / put it behind a login. But then don't complain that you are loosing customers because they can't see it or can not find information to your site through search engines.
> A better analogy would be me taking a photo of the bike while walking past.
Taking a photo of the bike is also a poor analogy. Scrapers don't take one photo, they take photos of all the bikes. And scrapers don't keep the photos for themselves, they sell them for a profit. Also, the original bike isn't parked, it's placed in a gallery (probably with an admission fee? I don't know the business model of linkedin).
But the owner is offering the images up for free to the public. hiQ is taking those publicly-available photos and annotating them with, "red bike", "pink bike", "broken bike", "professional bike", etc.
You have to keep in mind that an entire generation was brainwashed that personal data isn't that "personal", so Google, FB and the rest can have amazing profits.
Most of these discussions are stained by general unawareness of the privacy and copyright law.
Ofc because the value of the data supplier (usually a single person, etc) these never reaches the courts, which just reinforces the ongoing misconceptions.
If you really want to test this, try copying the content from Google, Facebook hiQ or whoever that's big enough to go after you.
But people somehow believe that it's okay for businesses to do what regular persons aren't allowed to.
The number of photos is irrelevant to the analogy, though, as is what people do with the photos afterwards. If the bikes are visible from the public street, people can take as many pictures of every bike they want, and then make money from them if they want. It doesn't affect the owners' usage of the bike (unlike the original analogy, where the owner loses access, which was what I was trying to correct)
Physical analogies for this kind of thing are always flawed, it's just dishonest/misleading to pretend that copying data is ever analogous to taking a physical object (the owner of the original is never deprived of the original when data is copied).
"I don't know the business model of linkedin"
Most of it is selling premium features to recruiters and other businesses. I'm not sure if Hi-Q's service interferes with that or not, but LinkedIn should not be trying to have their cake and eat it by leaving things in public then complaining when the public accesses it in a way they don't like.
> The number of photos is irrelevant to the analogy
Actually, the size of the data and the number of requests is very relevant. More data means more information, means more money. It also means more bandwidth and processing power required to process requests. You're not taking a photo of the bike, you're asking the bike to give you a photo of it.
> it's just dishonest/misleading to pretend that copying data is ever analogous to taking a physical object (the owner of the original is never deprived of the original when data is copied).
Leaving LinkedIn aside, possession of the original data is never the issue with digital piracy. It's a straw man. The hurt occurs when people benefit from the work the original author put into creating that data without proper compensation. Just because you can clone my gizmo (which I spent years working on) without taking the original one doesn't mean you're not hurting me. That gizmo could give me an advantage you wouldn't otherwise have. I place hours of working into something that doesn't put food on the table because you can clone my work, but I can't clone my food.
There's a reason an empty CD costs 50c but a music album costs $10. You're not paying for the physical medium. You're paying for the IP. And yes, digital distributions are cheaper because of this, but that doesn't make them free.
> Most of it is selling premium features to recruiters and other businesses.
I'd say it's pretty obviously interfering with their business model.
> LinkedIn should not be trying to have their cake and eat it by leaving things in public then complaining when the public accesses it in a way they don't like.
LinkedIn could ban IPs that make unreasonable number of requests in a short amount of time.
If LinkedIn are being that negatively affected by a single scraper, they should deal with it - block it, only allow a specific number of requests from an IP per day, anything that doesn't involve lawsuits. The problem is them trying to pretend that publicly visible content is really private if they say so, without them trying to protect it in any real way.
"The hurt occurs when people benefit from the work the original author put into creating that data without proper compensation"
Not necessarily. If I'm paying for print of some imaginative artwork that was created using the picture of the bike, that doesn't mean the bike owner lost anything, even if he spent time building the bike with his own hands. Similarly, if the only reason why people paid Hi-Q was for the extra work that they put in, LinkedIn didn't lose money because people would not have bought their product without that extra work.
There is certainly an argument that Hi-Q should have licenced the content first, but it's public data. If they want to make licence deals, don't put it in the view of the public street then whine when people are documenting what's in public.
"It's a straw man."
No, the straw man is pretending that a copy is the same as theft. Theft is theft because someone is depriving you of the original, not because you imagine you might have had more sales if the copy didn't exist. There's a reason why there are different words for different things, and pretending that a copy is the same as taking a physical object it a lie. Period.
"I place hours of working into something that doesn't put food on the table because you can clone my work, but I can't clone my food."
But, you put the price up too high, so I opted not to buy it. Maybe borrow the CD from a friend, or listen to something else. Or, you decided I couldn't buy it in the format or region I wanted. There are real issues, but pretending that a copy = a lost sale is utter bull that's been debunked time and time again, yet is regularly repeated by people trying to inject emotional arguments instead of facts.
"I'd say it's pretty obviously interfering with their business model"
Then perhaps they should address the business model or not put their content out there in public unprotected if it's that valuable to their income.
"LinkedIn could ban IPs that make unreasonable number of requests in a short amount of time."
Yes they could. Which would not have to involve the courts in any way. Or, they could protect the content in some other way that (for example) requires a log in and adherence to T&Cs, with which they could easily kick violators off their site for non-compliance.
The issue is that LinkedIn are trying to have it both ways - gathering the benefits of public content while blocking others who use the now-public content in ways that are usually acceptable for public content to be used. Sorry, not acceptable, you pick one - take the content away from the public street or accept that some people will use what has been shown to the public.
> If LinkedIn are being that negatively affected by a single scraper, they should deal with it - block it, only allow a specific number of requests from an IP per day, anything that doesn't involve lawsuits. The problem is them trying to pretend that publicly visible content is really private if they say so, without them trying to protect it in any real way.
With this, I agree 100%.
> No, the straw man is pretending that a copy is the same as theft. Theft is theft because someone is depriving you of the original, not because you imagine you might have had more sales if the copy didn't exist. There's a reason why there are different words for different things, and pretending that a copy is the same as taking a physical object it a lie. Period.
That's just pedantry. The debate isn't between "copy" and "theft", it's between "theft" and "copyright infringement".
> But, you put the price up too high, so I opted not to buy it. Maybe borrow the CD from a friend, or listen to something else. Or, you decided I couldn't buy it in the format or region I wanted. There are real issues, but pretending that a copy = a lost sale is utter bull that's been debunked time and time again, yet is regularly repeated by people trying to inject emotional arguments instead of facts.
This is wrong on so many levels, I'm not sure there's any point in continuing this debate. Are you accusing me of using emotional blackmail instead of facts because I point out that "you can clone my work, but I can't clone my food"?
I'm not using myself as an example because I want pity. I'm doing it because it's easier in writing, and because I'm a software developer.
My work takes hours of hours of time and effort (not accounting the hours I spent in school). If it' ok for everyone to clone my work, I won't make any money from it. We still live in a society where goods and services are exchanged with money. I exchanged my hours of work for no money, but I can't exchange no money for basic living necessities such as food. There's no feelings involved here. In the current economy, work going in, and no food coming out is not a viable business model. And if nobody payed for digital content, there would be a lot less digital content.
> pretending that a copy = a lost sale is utter bull that's been debunked time and time again
This is another straw man. Whether or not an illegal copy is or isn't a lost sale is irrelevant. You don't have the right to make that copy in the first place. If everyone made illegal copies, there would be no sales. So then why should only some be entitled to illegal copies? There isn't a distinction between people who can make copies and people who must pay for copies, so either everyone must pay for copies or no one must pay for copies. That's how law and economy work. You can't make exceptions by yourself. Either everyone is allowed, or no one is allowed. And for digital content that is for sale, no one is allowed illegal copies. If laws are made that allow poor people to receive goods for free, these laws must address both digital and physical goods.
What if the user is provided with permission to view medical records - your medical records for example? Would it be ok to scrape and sell those? What about children's school profiles and records?
Selling medical records would fall under a different law same goes for sharing credit #s cards you collect.
I think the issue is around public data here. If you made medical records available to the public you would be in trouble in many countries. Sharing those may land you in the same trouble as the website who initially posted it under that law. The person shouldn't be charged with unauthorized access because the materials are public.
It's more like complaining about others selling tickets to view said painting from the sidewalk. HiQ repackages and sells data it scrapes from LinkedIn.
The type of data being discussed here (factual data about people) cannot be copyrighted - i.e. the fact John Doe is a Software Engineer for ACME Inc is not copyrightable.
A better example I think is walking into a store and writing down what they price everything as, then selling aggregate pricing data to people. I can't see any reason that would be illegal.
Although I agree with you in principle, I think your analogy is flawed. The nature of a web request is that you're asking in the first place, and the serve has to serve you. If you're taking a photo, you're not asking anything; data transfer has a cost, too.
That is an interesting analogy. It reminds me of my experience visiting the Alamo in San Antonio. Once inside you cannot take photos. I cannot say for sure but I think there are armed guards that made sure cameras were not used.
Personally I think anything I can see I should be able to take a photo of for sentimental purposes.
Of course my original state was an over simplification, but I don't think it is like selling tickets, more like someone using a picture of the painting in the window to market their own products, since one person selling tickets to view the painting wouldn't do very well since they don't really 'block' the view for anyone else. I can still walk up right next to their queue and look at the painting. LinkedIn is arguing that no one should be allowed to walk by and take a picture, which is a right of panorama issue with all that that entails (in the US it would suggest that they don't have a case if they tried to use that argument by analogy).
edit: didn't see the sibling reply which makes the same point
LinkedIn and Reid Hoffman have stories (don't know Reid personally) that I don't buy as one of the first LinkedIn users, premium, ads, and rejected API customer. I think LinkedIn is a scam, where the service is done for the recruiters and job postings while they make you believe they are thinking in you. They never work for an OkCupid or FB like services but for businesses except adding basic introduction messaging.
> LinkedIn is a scam, where the service is done for the recruiters
Everyone knows that LinkedIn gets paid by recruiters rather than recruitees. But of course, that is perfectly fine. People are happy to be the product if it results in job leads.
This is not what they are selling with the premium offering and this why they are lying. They are offering a way to improve selling your products and services but without any technical innovation beyond messaging.
This is not what they are offering, look at the premium offerings: https://premium.linkedin.com/ do you know the ROI of using LinkedIn Premium (e.g. InMail) to just contact people using a zillion of methods available, and then adding them to your LinkedIn...
below the marketing language this is exactly those 2 points. guess it is easier if you work in enterprise, this business speak is a different language. pretty verbose, low information density.
Seems like Job websites like Indeed Prime and Google's new job product are serious competitors to LinkedIn.
LinkedIn is almost entirely worthless without the occasional good lead from a recruiter. I find it's mostly drowned out by the huge amounts of insignificant skin peddlers who dole out one worthless lead after another, looking for some fresh skin to peddle.
That law needs to be abused into the ground. We should all be filing frivolous lawsuits claiming CFAA violations whenever someone we don't like is accessing our sites. All we need is a little disclaimer in 5pt text stating who does not have authorization, like say all members of Congress.
Even if successfully litigated, doesn't this just move the scraping activity to less-obvious means, and to better-funded scrapers less concerned with legalities, making LinkedIn's efforts to clamp down upon scraping even more difficult? Is there a business case for LinkedIn to monetize the scraping by selling access to an API instead?
Between bot-nets, mechanical turks, deep learning, data brokering, lack of globally-enforced privacy laws that require divulging of sourced personal data, etc., I can't see a way for LinkedIn to prevent others from scraping and gaining from their publicly- and user-accessible data. They'll drive it underground, but if the concern is preventing others from grabbing the data at all, versus performance management, they'll still leak like a sieve.
This sounds like the setup to an up-spiraling arms race with "dark scrapers". I'm guessing behind-the-scenes, LinkedIn figures they can out-spend the extralegal scrapers, and likely considers their efforts will deliver halo effects to the rest of Microsoft. It would be educational to hear how LinkedIn plans to take down bot-net-based scraping that uses deep learning to identify patterns that successfully mimic human users and bypass their bot detection; could possibly help other white hats who want to battle bots and general malware.
If the goal of such an operation were to effectively create an alternative to LinkedIn, along the likes of other "claim your listing" sites, then this could be a worthwhile cause.
I'm skeptical of the potential of dark scrapers at scale. You'd need to simulate too much human behavior to be unidentifiable, and humans are slow.
You would need real-looking bot accounts that you'll use to scrape. You'd need a realistically randomized rate limit, sampling from some distribution conditional on the type of the source page. You'd need realistic mouse/keyboard movements. Realistic hours of operations. Can't be scraping at 4AM and 4PM, and all of the hours in-between. Occasional noise operations, such as searching for a job, or getting salary estimates. You'd be geographically constrained. You wouldn't want your bot from Boston to be looking at too many individuals in Houston (regularly). Maybe you'd use a Markov chain to have the bot make decisions? I doubt the blackhats would have good training data for a neural net. You'd need tens of thousands of these bots to cover the linkedin user base in reasonable time (say, once every week on average), and these bots would have to either overlap or seriously underlap on who they cover.
Best use case would scraper-API that you can use to look up batches of specific people, with your bots looking at others only to look realistic.
(Or maybe not? It's a fun question, but I know fuck all about this. Not my area of expertise.)
Along the lines of "fun question, I'll take a stab at it just for giggles"; this would be far more interesting as an interview question than "estimate how many soccer balls can fit in a 747".
Average botnet size is 20,000 compromised PC's. Srizbi is estimated at 450,000. Another vector I'd explore is teaming up with crypto-miners. As I understand it, there are no economic returns tapping into the CPUs any longer, so miners are using only GPUs and ASICs; if this is true, they'll have some spare CPU cycles, that they'd probably be willing to rent out to get some marginal returns on the CPUs that have to run and manage the mining chips, running a JVM or some other VM. If we can do that, then we can probably tap 2-3M hosts, many of them rotating in and out per day.
Throw out an army of mechanical turk assignments to get real humans to register fake accounts. They get paid upon submitting an account and password, which your scraping servers verify, then change the password and commandeer. Perhaps have them register the fake account while running under a container or VM on their computer; the container/VM is instrumented to capture all activity. The activity metrics and data are uploaded to a deep learning system, that identifies the patterns that work and the ones that don't, and uses that to guide the developers of what to randomize, and by how much.
Add in a component to randomly invite/follow other fake and real accounts, and generate Markov-chain-generated copypasta. Set aside a portion of the fake accounts to only build up networks of users. Initially restrict the market of customers to those who only want once-a-year-updated data. As the network builds, use the notification of changes to selectively scrape only changed user profiles, and upsell for more up-to-date profiles at that time.
If I was LinkedIn, I'd probably concentrate on infiltrating botnet operators, and shutting them down. It would be one large cat-and-mouse game.
""The CFAA makes it a crime to "access a computer without authorization or exceed authorized access." Courts have been struggling to figure out what this means ever since Congress passed it more than 30 years ago.""
The law by itself is ok, but I suspect lawmakers were referring to accessing a single personal workstation, probably not taking into account a cluster of servers containing public accessible data.
"Authorization" should be clarified to mean requiring credentials that formally grant access. In my eyes, public-facing content on a website is explicitly available to anyone. It clearly does not require authorization.
A perfect example of corporate hypocrisy. Both Facebook and LinkedIn did illegal things when they were younger but now that they are more established, they don't want anyone to do the same thing to them.
I mean that's just the standard way non altruistic entities interact with the law isn't it? Appeal to it when it benefits them, ignore it when it doesn't.
LinkedIn doesn't want YOUR work history easily shareable or consumed by other services. They sell this data to the highest bidders -- typically recruiters.
I'm seeing a lot of bad analogies thrown around in this comment thread, mostly based on emotional response and/or a dislike for LinkedIn.
As someone who has done a lot of scraping in the past (sometimes for good, sometimes not), the number one thing you need to respect as a scraper is that email or phone call you get saying "Stop doing that."
In almost all instances, you're legally fine in the real world until you get some communication to stop and/or blacklisted. After that point, what you are doing becomes a crime.
- LinkedIn is not a public resource, it is a private company that pays for servers.
- LinkedIn might scrape too, but that argument isn't going to hold up in court, and the scraping they do is probably in line with their EULA (protip: never install a social networking app on your phone, ever).
- The analogy to the storefront, taking pictures in public, etc, all break down because scraping LinkedIn requires you to access their resources.
- The analogy to browsing a store is great. If you are in a store, and they ask you to leave, and you don't, that's trespassing. Trespassing isn't legal.
The CFAA isn't a great law. There are a lot of gray areas. But LinkedIn seems within their rights here.
Congress writes bad laws all the time. So LinkedIn might be within their rights, but that doesn't mean they should have those rights.
It's bad for innovation to allow for selective discrimination like this. LinkedIn is perfectly happy to allow Google, Yahoo, Bing, and many, many more companies to scrape their content and use it for personal profit. Giving them the option to sue an upstart for doing exactly the same thing as Google is unfair and oligopolistic.
Letting established tech giants get away with this will slowly erode American dominance in technology.
> Giving them the option to sue an upstart for doing exactly the same thing as Google is unfair and oligopolistic.
But it's not. LinkedIn's data is their entire business. They are within their rights to restrict access to it.
This is the classic ant and grasshopper story. If HiQ wants access to the type of data they are scraping from LinkedIn, they can build that data themselves.
They are within their right to require authorization to their content, yes, but they don't do that. They make it public and allow some companies to scrape that content and resell it for profit (like Google), but while restricting other companies from doing the same.
If they don't want their data to be public, then they shouldn't make it public. They could require authorization to view any content on the site and solve this problem instantly.
I believe mywittyname is specifically referring to the LinkedIn profile pages that are publicly visible without requiring a login and is not claiming that all LinkedIn's data is public. Thus, the question is why doesn't LinkedIn simply hide all profile data behind user authentication?
> But it's not. LinkedIn's data is their entire business.
"There has grown in the minds of certain groups in this country the idea that just because a man or corporation has made a profit out of the public for a number of years, the government and the courts are charged with guaranteeing such a profit in the future, even in the face of changing circumstances and contrary to public interest. This strange doctrine is supported by neither statute or common law. Neither corporations or individuals have the right to come into court and ask that the clock of history be stopped, or turned back."
> If HiQ wants access to the type of data they are scraping from LinkedIn, they can build that data themselves.
The question is, how would you better define hacking? To make the letter of the law match the spirit? "Exceeding authorised access" is an early attempt that predates modern internet usage. You can't just say "anything the computer lets you do, is legal" because code exploits are just a computer following the (poorly-written) instructions in its code.
I'm not a lawyer, but it should a start with a clearly demonstrated attempt at prohibiting access to content. The words, "don't use this" are not enough. There needs to be active, ongoing safeguards to protect the data, i.e., authorization tokens, credentials, encryption keys, etc.
In the 80s, it was reasonable to assume that connecting to some port on a remote machine owned by another person or company could constitute unauthorized access. But today, billions of people connect to ports on remote machines thousands of times a day for completely legitimate reasons, so it's reasonable to assume that data that can be accessed by just asking nicely over the internet is considered intended for public consumption.
It seems permissive, but I think that's a crucial component. If some company makes accidentally makes their S3 buckets public, it's completely unfair to say that accessing that information is illegal, especially when they are serving up other information in public S3 buckets which they want people to access.
Many years ago, I wrote a scraper for a certain .mil financial website. My company at the time - and its customers - had legitimate accounts on that website, but the site was horrible to use. Page loads took upwards of a minute, and there were no front-page notification systems to let you know if any of the information on the site had changed. Kind of a big deal when it was a window into the government's accounts payable system. For instance, if you didn't notice that an auditor had left a comment on an invoice, and you didn't respond to it within X hours, then the gov't could deny the invoice and you'd have to start over. The end result was our customers paying full-time employees to manually examine every single open invoice in the .mil system every single day, which just increased load on the website and made it even slower.
Enter my scraper. It copied data into a local PostgreSQL database that our customers could run reports against. A process that used to take a human 6 hours a day now took 30 error-free seconds. The scraper was even a benefit to the website as we ran it overnight during low-load conditions, and because my software was smarter than a web browser, we could retrieve the same information as a human with about 1/3 the number of web requests. Perfect, right!
Well, no. We got an angry call from the developers complaining that we were the ones making their site slow, even though 1) we measurably didn't, 2) we were lighter on resources than the humans we were replacing, and 3) we scrupulously obeyed their rate limiting requirements and erred on the side of running 10% slower than they had originally requested from their customers.
That particular problem went away when we pointed out that their shiny new website wasn't Section 508 accessibility compliant and could not be made to be without literally throwing out their entire web service and starting over, but our website was, and that if they continued to allow us to serve screen reader compatible pages to our disabled customers then there wouldn't be a need to have the .mil website shut down and a Congressional investigation launched. All parties involved decided this was a reasonable compromise.
I've always believed that if you are scrapable then be prepared to be scraped. The internet is still a digital Wild West and stoping these guys will not stop some underground source with a cloud flare or tor hidden service or outside your countries laws from doing what they want with what they can get. they can build a site or they can sell it, stoping these guys just stops a public and honest company, sure they can stop the public companies but the best thing to do is protect yourself. Don't leave the keys to the door out front and get mad when someone uses the keys.
302 comments
[ 3.1 ms ] story [ 290 ms ] threadNot making a claim about scraping, but it's maybe more apt to compare to walking into a store and writing a list of everything for sale their costs.
cute.
I'm fine with that.
But I'm also fine if they do the like the article suggests and require anything non-scrapable to be behind an account prompt, even if everyone with a account can access it.
I don't think it's fair to make Linked In foot the bill for someone else's business. They shouldn't have to serve that content to people who aren't actually their users.
So put it behind a password. It's not reasonable to expect to get only the benefits of publicly-searchable data without any of the drawbacks.
If a Google-like competitor started, all Google would have to do to crush them is demand big-name sites formally prohibit the competitor from accessing their content or risk being delisted. And magically, it becomes impossible/illegal to build a duck-duck-go.
On one hand I'm against the cartel beahviour of Google doing something like that, but on the other hand, if Google asks and the other company agrees to block DDG, why shouldn't that be allowed?
And insurance companies only want to serve people who won't claim etc. LI is free to force users to usage agreements, but why should third-parties (taxpayers) foot the bill for enforcing them?
Anyone can make a LinkedIn account, ergo anyone can scrape any public profile.
The EFF said this, so they seem to agree that it set a bad precedent, even for those just scraping.
"Under the government's theory, anyone who disregards – or doesn't read – the terms of service on any website could face computer crime charges," said EFF civil liberties director Jennifer Granick in a press release at the time. "Price-comparison services, social network aggregators and users who skim a few years off their ages could all be criminals if the government prevails."
This is false. Web pages expect scrapers to respect robots.txt and users to respect the terms and conditions. From what I understand, the CFAA law was not meant for these violations, and the fight is whether or not this law can be used to win the trial. However, other laws can be put into place to address these situations.
I agree that what you say is the only sensible way to look at things.
If you have permission for one thing it doesn't mean you have permission for something different.
If you don't want it read, take it down.
First, the scraper consumes a lot more bandwidth (which some websites might have real trouble with). Second, the aggregated scraped data is a lot more valuable than what any single person receives from reading a few pages.
Websites could (and sometimes do) limit the number of views and the frequency of accesses from the same IP, which allows users to read the page and discourages scrapers without "taking it down".
"this scraping violated the Computer Fraud and Abuse Act, the controversial 1986 law that makes computer hacking a crime. HiQ sued, asking courts to rule that its activities did not, in fact, violate the CFAA."
Without going into too much detail, a lot of hedge funds have teams constantly searching for kernels of data that can contribute some kind of signal for market movements. This data can come in the form of satellite imagery for oil tankers or manufacturing centers, but it can also come from the very creative use of scraped and aggregated data. It's typically very difficult to identify, collect and analyze on a technical level (as 'chollida1 has lamented in the past: normalization, labeling/bucketing and analysis of disparate data across different formats, sources and processing timeframes is a pernicious problem at this scale). From a compliance standpoint there are also generally strict requirements governing legality of use.
Depending on the specific data, you might be capable of predicting earnings or broader market movements with a <5% margin of error each quarter for years at a time (I've personally seen and worked on projects with <1%, but that's the exception, not the norm). That tactic is usually found at discretionary funds; at quantitative funds the uses are much more abstract and cross-pollinated so as not to target single-equities, but rather holistic trends. Regardless, every fund is using data in some way these days; it's just a matter of how sophisticated, creative abstract they get in their analysis of it.
hiQ Labs doesn't collect data for this specific purpose, but it is absolutely related. In the past I have stayed away from crawling LinkedIn and Yelp precisely because they are very litigious (regardless of the eventual outcome and legality). Now that there's another relatively high profile case out in the open like this, I'm interested in seeing how it proceeds and what the ramifications will be for companies that collect data across a wide range of uses. As Grimmelman mentioned in the article, this can impact a lot of types of businesses, not just those in the same space as hiQ. Outside of finance I am familiar with many tech companies which (openly or otherwise), kickstarted what are now widely known enterprises through cleverly crawling or scraping massive amounts of data.
If hedge funds floating drones above oil tankers, I'd guess they'd be accussed of corporate espionage / spying / invasion of privacy?
Ok, so oil tankers are big and "in the clear". What if $TANKERCORP floats big parachute balloons above its tankers to imply "looking past these is unauthorized viewing"?
Then if a HedgeFund gets a clever angle on a satellite photo.. is that the equivalent of breaking a lock, or violating CFAA?
There are two other notes in response to your question:
1. Drones are different from satellites, and are more susceptible to regulation in the way you're positing because they can be prevented from flying above specific areas. However, most of the same problems with countering them apply, because drones can record better three dimensional footage. In your specific example, if a tanker disguised itself overhead, it would still be legal to have a drone monitor the tanker from the sides, as long as doing so didn't break any law set by the FAA.
2. Drones are actively used these days for things like monitoring production facilities ("how many cars come out of this factory" for an oversimplified version). If they have to monitor from a distance, so be it, they'll do it. The effective countermeasure here is to have a huge amount of land that can't provide any intelligence, because the drones aren't allowed to fly over it and can't see far enough in to the facility.
There's definitely a productive ethics discussion that can be had here, but the legal precedents don't really allow for combatting these techniques right now. If it's public, it can be collected, ingested and used in an algorithm to determine alpha.
In the Eastern European country from where I'm from (a NATO member) the Google StreetView car even got to photograph and publicly put on the Internet the outside of military and air bases with clear signs of "do not take photographs" visible on StreetView itself. It's funny, my company also used to work in this space (local business directory with business addresses, photos of said business etc) and one of my former colleagues got detained for a day because of taking photos of businesses in the downtown area of one of the biggest cities in my country. He hadn't seen that there was a military "objective" in his line of view (probably some military HQ or someth, not a proper military base with tanks and trucks). Talk about the advantages of being an internet giant like Google..
Later edit: I was talking for example about links like these: https://www.google.ro/maps/@44.4062748,26.0524843,3a,80.8y,2... . That is actually the HQ of NATO's "Multinational Division Southeast", whatever that is (http://www.nato.int/cps/in/natohq/news_125356.htm?selectedLo...). Fact is that if I were to take photos of those buildings as a simple citizen I would be breaking the law, not sure how Google got away with it.
It's not practical or safe to put a huge parachute or balloon over a tanker to block overhead imagery. Any sailor can tell you it just wouldn't work.
Google comes to my mind. I'm only partly joking, as it seems to me that the line between search engine web crawling and other forms of web scraping is very thin.
Are there "societies of scrapers"?
Inside, are certain sites more worthwhile - and which ones (eg reddit, eBay, trade union websites, whatever)
How about scraper brokers? Do they exist?
Are there scammy scrapers? Make up BS and sell as scraped data?
How big is this?
Not that I'm aware of, no.
> Inside, are certain sites more worthwhile - and which ones (eg reddit, eBay, trade union websites, whatever)
Yes, absolutely. For many purposes websites that sell their own data are less useful (less signal exclusivity). Specific sources of data will be much more valuable depending on what the data is about.
> How about scraper brokers? Do they exist?
Yes. You're not getting access without an NDA in addition to paying quite a lot.
> Are there scammy scrapers? Make up BS and sell as scraped data?
That depends on how easy it is to verify the data. For most of what you'd term "alternative data" you'll know if it's real in 2 - 12 weeks, and it's not sustainable to sell crap.
But a lot of parties scrape dodgy financial timeseries data (ticks and quotes on equities or options) and sell it, priced as though it were tick data when it's barely accurately OHLC. They mostly sell this sort of data to amateurs who don't realize tick data is expensive for a reason.
> How big is this?
Very big. Most hedge funds ingest a lot of data whether they curate it internally or source it from elsewhere.
If you scrape regularly, then pick up a dozen or more machines around the world, in less than friendly areas to US law. Pay with a rechargeable credit card or bitcoin. And the buy servers and set up a hadoop cluster that handles scan-jobs.
The worst case scenario is that LinkedIN, YELP, and others get some of your servers shut down. Wash, rinse, repeat.
EDIT: please note, this was only a thought-game to bypass rude and destructive laws like the CFAA, which weaponizes TOS'ses, EULA's, and other implicit contracts of adhesion (as in, you have to agree to see). Ideally, we would be better off with both these laws to have a sane scope, and for companies to not expect things to happen with content in public.
Given this is a predominantly a web development community it always surprises me how little creativity there is in the articles on investing. Neural networks and machine learning sound cool but the reality is almost none of the readership would be able to make any money off them.
Simply tracking how many sales or users exist in databases by watching sequential IDs should be the go-to method for any web developer trying to get an edge. I would have expected HN to have articles where people are getting creative on that, ie trying to use measures of entropy on usernames to get rough subscriber numbers etc.
Even plain scraping of prices etc, is often full of great insight that is ignored. If a grocery store drops their prices in profitable categories against their competitors, that could be the signal about an incoming price war for an entire sector. There's a lot more information in that than social media feeds and all the other sorts of sexy data that get coverage in the media.
Seems like a great application of the German tank problem [1] that was mentioned on HN the other day.
[1] https://en.wikipedia.org/wiki/German_tank_problem
If you look at the possible returns the equity market is going to make from a stock in a dollar value, and how much research spend is as a percentage of that, you'll quickly see it doesn't pay for much.
You can tell simply by looking at the broker research - that's probably the extent that analysts take things.
The big stocks obviously have a lot of it happening. (eBay listings, airline pricing etc is obviously touted a lot)
But once you start to go down to the mid caps, you enter a void where there isn't much heavy data focused research done, and it's very possible you can have a better gauge of the business than any other investor on the planet once you pull out this data out.
If you don't want it scraped, take it down, or put it behind a login.
If the user provides the login to a scraper, then the scraper has permission.
If I can walk near a pool, should I also be able to run? Is running anything more than faster walking? If I'm allowed to be around the pool walking with my entry ID, should I also be allowed to place my ID on a little motorized car and make it dart around the pool really fast? Should I be able to duplicate my badge, put it on a bunch of little cars and direct them to quickly get all the floaties before anyone else? How about giving them all their own fake IDs? Now all the same questions, except there is a sign that explicitly prohibits all of these examples except for walking.
It seems disingenuous to argue that the automation and rapid increase of a thing should be allowed just because a thing is allowed. That doesn't typically match our intuitive notions of ethics in other parts of society, like driving or walking around a pool. Yes, you can walk around a pool as much as you want, but if you change to running then you have fundamentally altered your behavior through increased capability, not merely done "more of walking" to utilize more of your freedom.
I suppose a natural counterargument to this analogy might be that running around pools is unsafe, and scraping is not unsafe in the same way. But my point here is establishing that a behavior intrinsically changes into a different behavior if you increase the speed at which you're doing it or the capability at which you can do it.
What does is it have to do with ethics? CFAA law does not apply here and that is what matters.
But if you contest that or have an issue with the specific terminology I used, perhaps you'd prefer this terminology instead: "I don't think this is a good framework for interpreting the CFAA and establishing legal precedent."
The analogy is starting to break down, but I think it's still instructive for the problem of applying a simple first principles approach.
Likewise in some high-crime jurisdictions, if you did not lock your car you are liable for it getting stolen or broken into[2]. An unlocked car is too tempting for some people to just walk past and not take it.
I know it might sound crazy but you could make the argument that a massive pool of highly-structured and very valuable data just sitting out in the open is an attractive nuisance and steps should be taken to put it behind a locked gate. Once that requirement has been satisfied, normal trespassing laws apply.
[1] https://en.wikipedia.org/wiki/Attractive_nuisance_doctrine
[2] http://www.cbc.ca/news/canada/montreal/ndg-resident-question...
For example, if a woman walks down a dark alley wearing short skirts and gets raped, it isn't her fault. I mean can you imagine if we said "well, she was just an attractive nuisance!" The judge would throw the book at you.
I am saddened by this.
Yes, that will cause potential issues for other people, which is why they tend not to do that, but if you trivially create a thousand more agents, and potentially trigger a degradation of service, how are you different to the people who block junctions at traffic lights?
I'm not keen on inconveniencing people, and "it's not that bad" is a poor argument for doing something that someone has explicitly asked you not to do.
That's why it might be reasonable for laws around this sort of thing to be different in the virtual world.
Once they have withdrawn permission, they can call the police and you can be charged with trespass, though that's usually a misdemeanor rather than a felony.
The nature of the original complaint is authorised access to publicly published information. Again, if you do not want people to read publicly published information, do not publish it publicly.
And as a side note - we don't need inappropriate analogies; the web is real, we can discuss the real issue.
Even if you go for "forgiveness rather than permission", if you ride it a second time after I've told you I don't want you riding it, then you're in the wrong.
Obviously, there are philosophical arguments to be made about the status of information online, but if the information were provided to LinkedIn, to be used on LinkedIn properties, I as a user would take umbrage with it being taken by a 3rd party, even if it were viewable publicly.
If you have an issue with that, you should be moving your bike somewhere people can't take a photo from the public street. Not have someone creatively interpret a law that says where I am is suddenly not public property, because you asked me to stop using my camera.
That would be copyright infringement because photographing a copyrighted work is considered reproducing the work under the law.
Perhaps my analogy wasn't great, but the grey area is around going to LinkedIn's server (whether or not this is "public" or their property they allow you access to is another philosophical question, though in the eyes of the law it appears it's the latter), deliberately extracting value from it, and then getting annoyed when you're asked not to.
Inherently it seems as though it's the old question of whether a server is public, or private but accessible (like those POPs [0] there was a thread on recently).
[0] https://en.wikipedia.org/wiki/Privately_owned_public_space
If it has those. The data on LI (other people's employment histories) is not its own IP.
Taking a photo of the bike is also a poor analogy. Scrapers don't take one photo, they take photos of all the bikes. And scrapers don't keep the photos for themselves, they sell them for a profit. Also, the original bike isn't parked, it's placed in a gallery (probably with an admission fee? I don't know the business model of linkedin).
It's clearly a value-add and not theft.
You have to keep in mind that an entire generation was brainwashed that personal data isn't that "personal", so Google, FB and the rest can have amazing profits.
Most of these discussions are stained by general unawareness of the privacy and copyright law.
Ofc because the value of the data supplier (usually a single person, etc) these never reaches the courts, which just reinforces the ongoing misconceptions.
If you really want to test this, try copying the content from Google, Facebook hiQ or whoever that's big enough to go after you.
But people somehow believe that it's okay for businesses to do what regular persons aren't allowed to.
Physical analogies for this kind of thing are always flawed, it's just dishonest/misleading to pretend that copying data is ever analogous to taking a physical object (the owner of the original is never deprived of the original when data is copied).
"I don't know the business model of linkedin"
Most of it is selling premium features to recruiters and other businesses. I'm not sure if Hi-Q's service interferes with that or not, but LinkedIn should not be trying to have their cake and eat it by leaving things in public then complaining when the public accesses it in a way they don't like.
Actually, the size of the data and the number of requests is very relevant. More data means more information, means more money. It also means more bandwidth and processing power required to process requests. You're not taking a photo of the bike, you're asking the bike to give you a photo of it.
> it's just dishonest/misleading to pretend that copying data is ever analogous to taking a physical object (the owner of the original is never deprived of the original when data is copied).
Leaving LinkedIn aside, possession of the original data is never the issue with digital piracy. It's a straw man. The hurt occurs when people benefit from the work the original author put into creating that data without proper compensation. Just because you can clone my gizmo (which I spent years working on) without taking the original one doesn't mean you're not hurting me. That gizmo could give me an advantage you wouldn't otherwise have. I place hours of working into something that doesn't put food on the table because you can clone my work, but I can't clone my food.
There's a reason an empty CD costs 50c but a music album costs $10. You're not paying for the physical medium. You're paying for the IP. And yes, digital distributions are cheaper because of this, but that doesn't make them free.
> Most of it is selling premium features to recruiters and other businesses.
I'd say it's pretty obviously interfering with their business model.
> LinkedIn should not be trying to have their cake and eat it by leaving things in public then complaining when the public accesses it in a way they don't like.
LinkedIn could ban IPs that make unreasonable number of requests in a short amount of time.
"The hurt occurs when people benefit from the work the original author put into creating that data without proper compensation"
Not necessarily. If I'm paying for print of some imaginative artwork that was created using the picture of the bike, that doesn't mean the bike owner lost anything, even if he spent time building the bike with his own hands. Similarly, if the only reason why people paid Hi-Q was for the extra work that they put in, LinkedIn didn't lose money because people would not have bought their product without that extra work.
There is certainly an argument that Hi-Q should have licenced the content first, but it's public data. If they want to make licence deals, don't put it in the view of the public street then whine when people are documenting what's in public.
"It's a straw man."
No, the straw man is pretending that a copy is the same as theft. Theft is theft because someone is depriving you of the original, not because you imagine you might have had more sales if the copy didn't exist. There's a reason why there are different words for different things, and pretending that a copy is the same as taking a physical object it a lie. Period.
"I place hours of working into something that doesn't put food on the table because you can clone my work, but I can't clone my food."
But, you put the price up too high, so I opted not to buy it. Maybe borrow the CD from a friend, or listen to something else. Or, you decided I couldn't buy it in the format or region I wanted. There are real issues, but pretending that a copy = a lost sale is utter bull that's been debunked time and time again, yet is regularly repeated by people trying to inject emotional arguments instead of facts.
"I'd say it's pretty obviously interfering with their business model"
Then perhaps they should address the business model or not put their content out there in public unprotected if it's that valuable to their income.
"LinkedIn could ban IPs that make unreasonable number of requests in a short amount of time."
Yes they could. Which would not have to involve the courts in any way. Or, they could protect the content in some other way that (for example) requires a log in and adherence to T&Cs, with which they could easily kick violators off their site for non-compliance.
The issue is that LinkedIn are trying to have it both ways - gathering the benefits of public content while blocking others who use the now-public content in ways that are usually acceptable for public content to be used. Sorry, not acceptable, you pick one - take the content away from the public street or accept that some people will use what has been shown to the public.
With this, I agree 100%.
> No, the straw man is pretending that a copy is the same as theft. Theft is theft because someone is depriving you of the original, not because you imagine you might have had more sales if the copy didn't exist. There's a reason why there are different words for different things, and pretending that a copy is the same as taking a physical object it a lie. Period.
That's just pedantry. The debate isn't between "copy" and "theft", it's between "theft" and "copyright infringement".
> But, you put the price up too high, so I opted not to buy it. Maybe borrow the CD from a friend, or listen to something else. Or, you decided I couldn't buy it in the format or region I wanted. There are real issues, but pretending that a copy = a lost sale is utter bull that's been debunked time and time again, yet is regularly repeated by people trying to inject emotional arguments instead of facts.
This is wrong on so many levels, I'm not sure there's any point in continuing this debate. Are you accusing me of using emotional blackmail instead of facts because I point out that "you can clone my work, but I can't clone my food"?
I'm not using myself as an example because I want pity. I'm doing it because it's easier in writing, and because I'm a software developer.
My work takes hours of hours of time and effort (not accounting the hours I spent in school). If it' ok for everyone to clone my work, I won't make any money from it. We still live in a society where goods and services are exchanged with money. I exchanged my hours of work for no money, but I can't exchange no money for basic living necessities such as food. There's no feelings involved here. In the current economy, work going in, and no food coming out is not a viable business model. And if nobody payed for digital content, there would be a lot less digital content.
> pretending that a copy = a lost sale is utter bull that's been debunked time and time again
This is another straw man. Whether or not an illegal copy is or isn't a lost sale is irrelevant. You don't have the right to make that copy in the first place. If everyone made illegal copies, there would be no sales. So then why should only some be entitled to illegal copies? There isn't a distinction between people who can make copies and people who must pay for copies, so either everyone must pay for copies or no one must pay for copies. That's how law and economy work. You can't make exceptions by yourself. Either everyone is allowed, or no one is allowed. And for digital content that is for sale, no one is allowed illegal copies. If laws are made that allow poor people to receive goods for free, these laws must address both digital and physical goods.
If there is a problem with that, see the second line
I think the issue is around public data here. If you made medical records available to the public you would be in trouble in many countries. Sharing those may land you in the same trouble as the website who initially posted it under that law. The person shouldn't be charged with unauthorized access because the materials are public.
It's more like complaining about others selling tickets to view said painting from the sidewalk. HiQ repackages and sells data it scrapes from LinkedIn.
It's like taking pictures of the paintings from the street, and reselling those pictures.
If they don't like that... that the painting down. Simple.
https://en.wikipedia.org/wiki/Sui_generis_database_right
As with all rights, it varies with jurisdiction.
Personally I think anything I can see I should be able to take a photo of for sentimental purposes.
edit: didn't see the sibling reply which makes the same point
Everyone knows that LinkedIn gets paid by recruiters rather than recruitees. But of course, that is perfectly fine. People are happy to be the product if it results in job leads.
1., Self-updating rolodex for sales people. 2., Recruitment tool.
It works for both, no real competitor in sight due to the massive network effect.
Some niche networks are doing ok, like Xing in DACH region. Not aware of anything special in China, guess everyone is on WeChat anyhow.
below the marketing language this is exactly those 2 points. guess it is easier if you work in enterprise, this business speak is a different language. pretty verbose, low information density.
LinkedIn is almost entirely worthless without the occasional good lead from a recruiter. I find it's mostly drowned out by the huge amounts of insignificant skin peddlers who dole out one worthless lead after another, looking for some fresh skin to peddle.
Between bot-nets, mechanical turks, deep learning, data brokering, lack of globally-enforced privacy laws that require divulging of sourced personal data, etc., I can't see a way for LinkedIn to prevent others from scraping and gaining from their publicly- and user-accessible data. They'll drive it underground, but if the concern is preventing others from grabbing the data at all, versus performance management, they'll still leak like a sieve.
If your business is in the recruiter space, expect to have your API keys revoked and to receive a cease and desist letter as well.
You would need real-looking bot accounts that you'll use to scrape. You'd need a realistically randomized rate limit, sampling from some distribution conditional on the type of the source page. You'd need realistic mouse/keyboard movements. Realistic hours of operations. Can't be scraping at 4AM and 4PM, and all of the hours in-between. Occasional noise operations, such as searching for a job, or getting salary estimates. You'd be geographically constrained. You wouldn't want your bot from Boston to be looking at too many individuals in Houston (regularly). Maybe you'd use a Markov chain to have the bot make decisions? I doubt the blackhats would have good training data for a neural net. You'd need tens of thousands of these bots to cover the linkedin user base in reasonable time (say, once every week on average), and these bots would have to either overlap or seriously underlap on who they cover.
Best use case would scraper-API that you can use to look up batches of specific people, with your bots looking at others only to look realistic.
(Or maybe not? It's a fun question, but I know fuck all about this. Not my area of expertise.)
Average botnet size is 20,000 compromised PC's. Srizbi is estimated at 450,000. Another vector I'd explore is teaming up with crypto-miners. As I understand it, there are no economic returns tapping into the CPUs any longer, so miners are using only GPUs and ASICs; if this is true, they'll have some spare CPU cycles, that they'd probably be willing to rent out to get some marginal returns on the CPUs that have to run and manage the mining chips, running a JVM or some other VM. If we can do that, then we can probably tap 2-3M hosts, many of them rotating in and out per day.
Throw out an army of mechanical turk assignments to get real humans to register fake accounts. They get paid upon submitting an account and password, which your scraping servers verify, then change the password and commandeer. Perhaps have them register the fake account while running under a container or VM on their computer; the container/VM is instrumented to capture all activity. The activity metrics and data are uploaded to a deep learning system, that identifies the patterns that work and the ones that don't, and uses that to guide the developers of what to randomize, and by how much.
Add in a component to randomly invite/follow other fake and real accounts, and generate Markov-chain-generated copypasta. Set aside a portion of the fake accounts to only build up networks of users. Initially restrict the market of customers to those who only want once-a-year-updated data. As the network builds, use the notification of changes to selectively scrape only changed user profiles, and upsell for more up-to-date profiles at that time.
If I was LinkedIn, I'd probably concentrate on infiltrating botnet operators, and shutting them down. It would be one large cat-and-mouse game.
The law by itself is ok, but I suspect lawmakers were referring to accessing a single personal workstation, probably not taking into account a cluster of servers containing public accessible data.
They have no legal obligation to provide accurate information to scrapers.
* What if someone had a few oDesk workers to manually enter in the LinkedIn data?
* What if someone wrote a scraper that parses the Google cache version of the public LinkedIn page?
As someone who has done a lot of scraping in the past (sometimes for good, sometimes not), the number one thing you need to respect as a scraper is that email or phone call you get saying "Stop doing that."
In almost all instances, you're legally fine in the real world until you get some communication to stop and/or blacklisted. After that point, what you are doing becomes a crime.
- LinkedIn is not a public resource, it is a private company that pays for servers. - LinkedIn might scrape too, but that argument isn't going to hold up in court, and the scraping they do is probably in line with their EULA (protip: never install a social networking app on your phone, ever). - The analogy to the storefront, taking pictures in public, etc, all break down because scraping LinkedIn requires you to access their resources. - The analogy to browsing a store is great. If you are in a store, and they ask you to leave, and you don't, that's trespassing. Trespassing isn't legal.
The CFAA isn't a great law. There are a lot of gray areas. But LinkedIn seems within their rights here.
If anyone wants to know how this is going to wind up: https://www.eff.org/deeplinks/2015/06/padmapper-and-3taps-se...
Congress writes bad laws all the time. So LinkedIn might be within their rights, but that doesn't mean they should have those rights.
It's bad for innovation to allow for selective discrimination like this. LinkedIn is perfectly happy to allow Google, Yahoo, Bing, and many, many more companies to scrape their content and use it for personal profit. Giving them the option to sue an upstart for doing exactly the same thing as Google is unfair and oligopolistic.
Letting established tech giants get away with this will slowly erode American dominance in technology.
But it's not. LinkedIn's data is their entire business. They are within their rights to restrict access to it.
This is the classic ant and grasshopper story. If HiQ wants access to the type of data they are scraping from LinkedIn, they can build that data themselves.
If they don't want their data to be public, then they shouldn't make it public. They could require authorization to view any content on the site and solve this problem instantly.
"There has grown in the minds of certain groups in this country the idea that just because a man or corporation has made a profit out of the public for a number of years, the government and the courts are charged with guaranteeing such a profit in the future, even in the face of changing circumstances and contrary to public interest. This strange doctrine is supported by neither statute or common law. Neither corporations or individuals have the right to come into court and ask that the clock of history be stopped, or turned back."
> If HiQ wants access to the type of data they are scraping from LinkedIn, they can build that data themselves.
https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....
In the 80s, it was reasonable to assume that connecting to some port on a remote machine owned by another person or company could constitute unauthorized access. But today, billions of people connect to ports on remote machines thousands of times a day for completely legitimate reasons, so it's reasonable to assume that data that can be accessed by just asking nicely over the internet is considered intended for public consumption.
It seems permissive, but I think that's a crucial component. If some company makes accidentally makes their S3 buckets public, it's completely unfair to say that accessing that information is illegal, especially when they are serving up other information in public S3 buckets which they want people to access.
Enter my scraper. It copied data into a local PostgreSQL database that our customers could run reports against. A process that used to take a human 6 hours a day now took 30 error-free seconds. The scraper was even a benefit to the website as we ran it overnight during low-load conditions, and because my software was smarter than a web browser, we could retrieve the same information as a human with about 1/3 the number of web requests. Perfect, right!
Well, no. We got an angry call from the developers complaining that we were the ones making their site slow, even though 1) we measurably didn't, 2) we were lighter on resources than the humans we were replacing, and 3) we scrupulously obeyed their rate limiting requirements and erred on the side of running 10% slower than they had originally requested from their customers.
That particular problem went away when we pointed out that their shiny new website wasn't Section 508 accessibility compliant and could not be made to be without literally throwing out their entire web service and starting over, but our website was, and that if they continued to allow us to serve screen reader compatible pages to our disabled customers then there wouldn't be a need to have the .mil website shut down and a Congressional investigation launched. All parties involved decided this was a reasonable compromise.