tl;dr: FBI wants to have its own version of what every startup that scrapes data from social media sites have been doing already, and the only change is that "keywords" and "trends" are now called "threats".
I will start worrying when companies are legally required to let the FBI trawl through the databases without a warrant. It's hard to make a law against scraping public information.
Someone else raised this point elsewhere but...so what? Monitoring public information is their job. I'd be disappointed if they weren't monitoring those services.
If you post public information, it's publicly consumable, including by government intelligence services. If you have a problem with that, consider that the issue might be your choice to post information publicly that you don't want the FBI to see, rather than the fact that the FBI is looking at it.
The difference as far as I can see is that as a startup would I be granted access to all the data? AFAIK facebook terms say I cannot scrape the information and in the past they had forced a hacker to delete the database that was collected by scraping this info.
This sets a dangerous precedent. By letting a government authority to oversee the activity, even if considered public, is still personal. Its a step closer to the cliche - "In Soviet Russia TV watches you"
Correct, there is no indication of that. I would be really happy to see if Facebook/twitter reject FBI's request. I was just hypothesizing that in the event that FBI is granted scrapping access, they should give similar access to other bots (or even Google to download contacts?).
The ideal situation would be that FBI is not granted access.
The terms restrict scraping without permission. I'm pretty sure that the FBI would be able to find a way to get Fb's permission. And for good reason. Again, it's public information. No harm done.
It is fine as long as other companies/startups are granted such access. The problem here is if FBI can wield a bigger power in getting this access, we have a problem. Not so much with the data itself, but the spirit of how someone with power can get their way.
I think there's a slight difference in that you have an expectation of ephemerality in real-world conversations that you do not (or at least, should not) with Facebook et al.
This isn't like the FBI setting up surveillance equipment to capture and monitor conversations; it's like the FBI asking to look at the tapes that people are already gleefully making and leaving around in public bins for others to pick up and watch.
I get the reasoning - and honestly, it was my first instinct on the news as well - but I don't think it's a direct analog.
I figured they were already doing this. Once I heard that Library of Congress was archiving tweets I assumed they either were or would be monitoring that information. I always heard that they have computers listening for keywords on phone conversations also. I never knew if it was true or a urban legend.
I can't imagine the amount of channel noise generated by certain keywords. And for every one you'd have to do a time consuming search through the user's social graph(s).
There should be a function in the scraper that gives each user a unique ID, and retains an instance count of keyword 'hits'. Over time, the user would build up a 'heat rating' and IDs could be sorted by this. That way, people generating keyword noise over and over again could be separated from the general population for a closer look.
We all act as peers in a large social structure in the current world. I have heard about the idea of P2P social networks several times. However, the trend seem to be going in the opposite direction: Look at the cloud.
I would think the majority of Facebook users would not see the need for having an always-on server in their own home hosting their social network, and therefore have a hard time imagining that "P2P social [networking] is the future". They would either have to have this in their own home, or rented from some company, costing money, and, once again, centralising information.
Does the fact they specifically include MySpace in Section H, subsection a, in the original Request for Information indicate just how disconnected the people who are writing these are from consumer internet trends?
They have already been doing it for a few years now. I don't think you understand what sort of infiltration software their contractors make. It gets deep fast. They aren't as good as some of the stuff at other three letter agencies, but its getting there.
If only you guys on the West coast knew what gets built here on the East coast...
This seems to be very clearly against Facebook and Twitter's terms, unless they are given permission:
Facebook:
"You will not collect users' content or information, or otherwise access Facebook, using automated means (such as harvesting bots, robots, spiders, or scrapers) without our permission."
- https://www.facebook.com/legal/terms
Twitter:
"You may not do any of the following while accessing or using the Services: (i) access, tamper with, or use non-public areas of the Services, Twitter’s computer systems, or the technical delivery systems of Twitter’s providers; (ii) probe, scan, or test the vulnerability of any system or network or breach or circumvent any security or authentication measures; (iii) access or search or attempt to access or search the Services by any means (automated or otherwise) other than through our currently available, published interfaces that are provided by Twitter (and only pursuant to those terms and conditions), unless you have been specifically allowed to do so in a separate agreement with Twitter (NOTE: crawling the Services is permissible if done in accordance with the provisions of the robots.txt file, however, scraping the Services without the prior consent of Twitter is expressly prohibited)"
- https://twitter.com/tos
I'm more interested in seeing what they end up paying for this software when it's built. I'm sure the right couple of geeks from HN could do it in a few months for fairly little; I'm sure the FBI will end up paying many millions.
The FBI would be smart to learn from DARPA and take a prize approach on projects like this. "First company that builds X to our satisfaction will get $1mm." I imagine it'd save them a ton.
38 comments
[ 4.0 ms ] story [ 78.3 ms ] threadhttps://www.fbo.gov/utils/view?id=7f9abf0ff0fdba171d1130ddf4...
http://www.youtube.com/watch?v=cqggW08BWO0
I will start worrying when companies are legally required to let the FBI trawl through the databases without a warrant. It's hard to make a law against scraping public information.
If you post public information, it's publicly consumable, including by government intelligence services. If you have a problem with that, consider that the issue might be your choice to post information publicly that you don't want the FBI to see, rather than the fact that the FBI is looking at it.
This sets a dangerous precedent. By letting a government authority to oversee the activity, even if considered public, is still personal. Its a step closer to the cliche - "In Soviet Russia TV watches you"
Congratulations: you are now living in a police state.
"Seeing it" because it's public and "monitoring" it systematically are two very different things.
This isn't like the FBI setting up surveillance equipment to capture and monitor conversations; it's like the FBI asking to look at the tapes that people are already gleefully making and leaving around in public bins for others to pick up and watch.
I get the reasoning - and honestly, it was my first instinct on the news as well - but I don't think it's a direct analog.
If they had been given access to private posts, that would be a cause for concern.
There should be a function in the scraper that gives each user a unique ID, and retains an instance count of keyword 'hits'. Over time, the user would build up a 'heat rating' and IDs could be sorted by this. That way, people generating keyword noise over and over again could be separated from the general population for a closer look.
I would think the majority of Facebook users would not see the need for having an always-on server in their own home hosting their social network, and therefore have a hard time imagining that "P2P social [networking] is the future". They would either have to have this in their own home, or rented from some company, costing money, and, once again, centralising information.
If only you guys on the West coast knew what gets built here on the East coast...
Facebook: "You will not collect users' content or information, or otherwise access Facebook, using automated means (such as harvesting bots, robots, spiders, or scrapers) without our permission." - https://www.facebook.com/legal/terms
Twitter: "You may not do any of the following while accessing or using the Services: (i) access, tamper with, or use non-public areas of the Services, Twitter’s computer systems, or the technical delivery systems of Twitter’s providers; (ii) probe, scan, or test the vulnerability of any system or network or breach or circumvent any security or authentication measures; (iii) access or search or attempt to access or search the Services by any means (automated or otherwise) other than through our currently available, published interfaces that are provided by Twitter (and only pursuant to those terms and conditions), unless you have been specifically allowed to do so in a separate agreement with Twitter (NOTE: crawling the Services is permissible if done in accordance with the provisions of the robots.txt file, however, scraping the Services without the prior consent of Twitter is expressly prohibited)" - https://twitter.com/tos
(edit: typo)
The FBI would be smart to learn from DARPA and take a prize approach on projects like this. "First company that builds X to our satisfaction will get $1mm." I imagine it'd save them a ton.
Duplex scanner.
Like, does anyone think that they aren't already?
[1] The current "Download my Data" functionality is a joke. Try it for yourself and see what I mean.