He crawled a bunch (780 Gigs / ~20 million pages) of 'public' court records that were behind some kind of auth (pay) wall via a free trial and opened them up. Likely as a result of this crawling, the free trial was shut down. The pages he crawled constitute 20% of the total records. The feds then started investigating him.
Court records are, in general, public. Not 'public'. Just go to the library. However, PACER costs money and so do the favored online vendors (westlaw and lexis-nexis).
Even if they aren't public. They should be. This stuff can be used as law -- precedents. The only valid objection I see is that there needs to be some process to protect the privacy of the people involved in the case. But that's second-order concern.
PACER records are public, even that they cost money to access. So are most of the records on Lexis and Westlaw. Westlaw has tried to claim a copyright in certain annotations on the court documents, but they have no copyright in the text of the court records themselves.[1]
I don't know much about what PACER includes and what exactly is covered by Westlaw and Lexis-Nexis, but Altlaw (http://altlaw.org) is a "free, full-text, searchable database of U.S. case law", and I've heard it being mentioned in the context of Lexis-Nexis. Also has the extra cool aspect of having been done by Stuart Sierra, whom all the Clojure folks around here probably know well.
But when lawyers told Mr. Malamud and Mr. Swartz that they appeared to have broken no laws, Mr. Malamud sent Mr. Swartz a message saying, “You should just lay low for a while.”
Either they think it is illegal or what is more likely is that anytime a sysadmin at a federal facility sees "atypical" behavior (as defined by them) they contact the feds. It is very unlikely that a sysadmin in the federal court system would think about whether the crawling session overloading his server is legal, forget moral and I am sure if he/she complains, the FBI is obligated to investigate.
I am not saying this is what should happen but this is what likely happened.
Interpretation of 'unauthorized access' is murky and there's a lot -- too much -- discretion in how access legitimacy and damages are evaluated. (See also: the submission last week about an author's estimation most people commit 'three felonies a day'.)
In this matter, I'd bet the NYTimes coverage was more valuable in deterring prosecution than any legal counsel or even influential friends -- it meant any trumped-up narrative of nefarious intent would be fighting an uphill battle against the story already started in 'the paper of record'.
PACER data is now being collected and made available to the public by the RECAP project run by the Center for Information Technology Policy at Princeton.
House is set on a deep lot, behind other houses on Marshman Avenue. This is a heavily wooded, dead-end street, with no other cars parked on the road making continued surveillance difficult to conduct without severely increasing the risk of discovery.
Good thing it was just the FBI. The BATF might have classified this mysterious residence as some sort of cult compound and executed a no-knock raid with automatic weapons drawn.
funny, this reminds me of the typical series of bugzilla comments i make when I'm debugging something. although this is a lot more detailed and thorough than I ever am...
and guess there is opportunity here. FBI agents can be excellent trainers for issue tracking and documenting projects. One of them should start a blog.
Aaron, how did you know on Feb 13 that the FBI had opened a file on you? From my skim, the first contact was April 14, or was there something else before then?
The two accounts were responsible for downloading more than
eighteen million pages with an approximate value of $1.5 million.
Ha! Just because they charge $0.08 a page doesn't mean millions of page views online cost the same amount.
This reminds of big drug catches that bring in tons of a controlled substance. The value is always given in street value, to make the efforts seem more substantial. Too bad they never publicize stats that show the police and FBI catch an infinitesimally small percentage of narcotics.
yeah, but I thought that's how judicial system in the US works - they add listed value of every single stolen/smuggled/whatever piece without any respect of mark-to-market for the whole batch?
and that's kind of understandable - court has no time/expertise/willingness to figure out what "market" value of stuff should be.
Yes, but if they determine punishment based on the value of 'goods' 'stolen', then hopefully they would get an accurate estimate of their value, not values inflated by law enforcement.
The value of the 'stolen goods' here is easy to peg - it would have cost $.08 per page to obtain it legally. Whether or not the item is "worth" the price charged is not a matter for the court to decide.
But in another way you're right - piracy is closer to counterfeit than theft. The original item is still available to its rightful owner to sell, although the copy arguably dilutes its value.
(An old roommate of mine got busted by the FBI for large-scale piracy, so I'm more familiar with it than I want to be.)
The Administrative Office of the U.S. Courts reported that the PACER system was being inundated with requests. One request was being made every three seconds.
Inundated by one request every three seconds? That's pretty pathetic if you ask me. They must not have planned for the PACER system to be used very much by very many people.
It depends on what you're using it for. Unless you need the bleeding edge of new stories, 30-45 minutes between each scrape is probably fine. If you're looking to stay on top of 'new', your best bet is about every 10 minutes.
More frequent than that, and you're just causing trouble for the server with no real benefit for yourself.
The two accounts were responsible for downloading more than eighteen million pages with an approximate value of $12. and estimated profit loss of $1.499988 million.
"... NCIS report for Aaron Swartz was negative ..."
excellent.
the other thing I don't get is flags should be going up if nothing bad is being turned up. as for being legal? I bet Pacer didn't think/want 20% of their records searched even if they opened them up. It's unstated but it would be fair to assume this.
From what I've read here they (fbi) are doing simple background checks and want to talk. An interview/Talk doesn't imply a presumption of guilt. Having said that the fact that the "pacer" usage may have been excessive & someone has complained means anyone with excessive usage is going to be looked at.
There is no black and white here but shades of grey. The records probably should be in public hands. The means used so far might be illegal. The tussle is "public access" vs "payment" and the action taken is making the right waves to hightlight the need for the former ~ http://www.nytimes.com/2009/02/13/us/13records.html
"... In his call to action, Mr. Malamud pointed to the free trial Pacer was offering and called for a “Thumb Drive Corps” to go to libraries with small-but-capacious “thumb drives,” plug them into computers, download as many court documents as they could, and send them to Mr. Malamud so that he could translate them them into a format that Google’s search software can read and put them on line. ..."
Bill Waterson's Calvin & Hobbes poem introduced "defenestration" to me when I was young young young: "The monster, in his consternation,
Demonstater defenestration". Its such a great word, and he was such a brilliant person, and a great inspiration. (sorry, completely off topic)
I'm mostly fascinated by their attempt to trick him to give up compromising information about his actions, by asking him for help on how to secure the system. If they would've asked me before reading this, I would've probably been too naive to ask for assurance that what I said wouldn't be used against me.
Yeah, the police are good at this. I was investigated by the Secret Service once. Long story involving a sarcastic non-anonymous slashdot comment that was misinterpreted as a threat on the President's life. (Yeah, America is a strange place.)
Anyway, the agents came to my house, made a vague accusation, and asked me if I knew anything about this. Having watched a bit of Law & Order, my answer was, "I think I need to talk to my lawyer". They persisted, and provided me with a lot of false information. At this point, I was annoyed at how wrong they were and said "yeah, I wrote this" and pulled up the comment in question.
(Resolution: they read the comment, realized that someone Higher Up totally overreacted, and went away.)
Basically, detectives are trained to extract information, and they are good at doing so. (Worth noting that after they got the information they wanted, they were generally pleasant people. Real life can be very much like television. :)
I really wish I knew this information about 16 years ago when I was still unbelievably naive. I was raised to answer police officers honestly, and to assume that they were also honest. And I still thought our criminal justice system worked properly and everyone was actually interested in justice. My first lesson that the world wasn't like Mayberry was a bit harsh.
Mr. Swartz is known to frequent "Hacker News," an underground forum operated by the "partner in crime" of convicted computer criminal Robert Tappan Morris.
Based on the FBI's discovery of the subject's Facebook and LinkedIn profiles, they haven't yet found the subject's interest in the underground BBS known as 'Hacker News'.
I had to request my FBI file once, and there was none. But the papers they send you are woefully misleading. It says something to the effect of 'Below are all the documents:' and under the rest of the page is blank. A 'None exist' would have been a little helpful. I kept thinking that they forgot to mail me some other page, but I was able to get by with what they gave me.
IIRC the cost was $18. But don't send a personal cheque, or you'll get a call from some FBI processing person directing you to fax them credit card authorization or to completely refile the request. Mine took like 7 or 8 months to process. It was 3 or 4 months after I sent off the request before I got the call about the issue with payment through a personal cheque, which gives you an idea of what the backlog was probably like. This was around 2 years ago.
Not sure what the point of this comment is. Fingerprints are one of the requirements for requesting any law enforcement records (not court documents) so far as I know.
It's not like this is some sort of insidious process where you don't realize they are going to take your fingerprints until it's too late for you to back out because there is a gun against your head. The entire process is basically:
1. Get fingerprints on a card of acceptable format by a 'professional.'
2. Send in fingerprint card, request and payment.
As for me, it's hard to go through the immigration process without proving that you're not a criminal in your country of origin. That, and I have my fingerprints on record anyways. I had a class 3 gaming license in NY state. The NY State Police took my fingerprints for that.
I'm a UK citizen and I apparently have none, because I requested for me and Don Simpson (the deceased film producer), and they only said 'No file on Don Simpson' - didn't even say 'No file on [Zandorg]'.
88 comments
[ 4.8 ms ] story [ 160 ms ] threadhttp://thelede.blogs.nytimes.com/2009/02/13/steal-these-fede...
Even if they aren't public. They should be. This stuff can be used as law -- precedents. The only valid objection I see is that there needs to be some process to protect the privacy of the people involved in the case. But that's second-order concern.
[1] Matthew Bender & Co. v. West Publishing Co., 158 F.3d 693 (2d Cir. 1998) - http://bulk.resource.org/courts.gov/c/F3/158/158.F3d.693.97-...
There already is: you can file documents under seal or in redacted form. Most courts require you to do so if you're including information such as social security numbers. See, i.e. http://www.courtinfo.ca.gov/invitationstocomment/documents/s...
Nothing that's available via PACER couldn't be read by walking down to the courthouse and requesting the physical file.
It's not illegal, so what's going on?
Trying to collect $1.5MM?
I am not saying this is what should happen but this is what likely happened.
In this matter, I'd bet the NYTimes coverage was more valuable in deterring prosecution than any legal counsel or even influential friends -- it meant any trumped-up narrative of nefarious intent would be fighting an uphill battle against the story already started in 'the paper of record'.
https://www.recapthelaw.org/how-it-works/
Good to know.
This reminds of big drug catches that bring in tons of a controlled substance. The value is always given in street value, to make the efforts seem more substantial. Too bad they never publicize stats that show the police and FBI catch an infinitesimally small percentage of narcotics.
and that's kind of understandable - court has no time/expertise/willingness to figure out what "market" value of stuff should be.
But in another way you're right - piracy is closer to counterfeit than theft. The original item is still available to its rightful owner to sell, although the copy arguably dilutes its value.
(An old roommate of mine got busted by the FBI for large-scale piracy, so I'm more familiar with it than I want to be.)
The Administrative Office of the U.S. Courts reported that the PACER system was being inundated with requests. One request was being made every three seconds.
Inundated by one request every three seconds? That's pretty pathetic if you ask me. They must not have planned for the PACER system to be used very much by very many people.
More frequent than that, and you're just causing trouble for the server with no real benefit for yourself.
there. fixed it for ya.
excellent.
the other thing I don't get is flags should be going up if nothing bad is being turned up. as for being legal? I bet Pacer didn't think/want 20% of their records searched even if they opened them up. It's unstated but it would be fair to assume this.
From what I've read here they (fbi) are doing simple background checks and want to talk. An interview/Talk doesn't imply a presumption of guilt. Having said that the fact that the "pacer" usage may have been excessive & someone has complained means anyone with excessive usage is going to be looked at.
There is no black and white here but shades of grey. The records probably should be in public hands. The means used so far might be illegal. The tussle is "public access" vs "payment" and the action taken is making the right waves to hightlight the need for the former ~ http://www.nytimes.com/2009/02/13/us/13records.html
There's a bigger picture (activism) here & maybe that's why the fbi has some interest. Having read this article ~ http://www.nytimes.com/2009/02/13/us/13records.html I can now see why.
http://dictionary.reference.com/browse/exfiltrated
Next time we should coordinate, and pull a "Spartacus".
EDIT: Apparently your webserver is a little less resilient though. :-p
Anyway, the agents came to my house, made a vague accusation, and asked me if I knew anything about this. Having watched a bit of Law & Order, my answer was, "I think I need to talk to my lawyer". They persisted, and provided me with a lot of false information. At this point, I was annoyed at how wrong they were and said "yeah, I wrote this" and pulled up the comment in question.
(Resolution: they read the comment, realized that someone Higher Up totally overreacted, and went away.)
Basically, detectives are trained to extract information, and they are good at doing so. (Worth noting that after they got the information they wanted, they were generally pleasant people. Real life can be very much like television. :)
[just in case there is someone who hasn't seen this before]
IIRC the cost was $18. But don't send a personal cheque, or you'll get a call from some FBI processing person directing you to fax them credit card authorization or to completely refile the request. Mine took like 7 or 8 months to process. It was 3 or 4 months after I sent off the request before I got the call about the issue with payment through a personal cheque, which gives you an idea of what the backlog was probably like. This was around 2 years ago.
It's not like this is some sort of insidious process where you don't realize they are going to take your fingerprints until it's too late for you to back out because there is a gun against your head. The entire process is basically:
1. Get fingerprints on a card of acceptable format by a 'professional.'
2. Send in fingerprint card, request and payment.
As for me, it's hard to go through the immigration process without proving that you're not a criminal in your country of origin. That, and I have my fingerprints on record anyways. I had a class 3 gaming license in NY state. The NY State Police took my fingerprints for that.
:) I wonder how they will deal with more and more hacker-type people of independent means.