It certainly gives you a feel for how they do development internally. End of the readme mentions a new version coming soon, instead of say, iterating on this one. Also, the main contributor is stripped of any personally identifiable information.
Interesting to get open source from the British sort-of equivalent to the NSA.
I just looked at the code: Java code that sits on top of the Hadoop file system. Supports date-binned data storage so it looks applicable to systems where you want to toss out old data occasionally.
Every oppressive regime's secret apparatus is basically a large scale database management system. Whether that's with files and folders a la Stasi or with huge data centers in Utah...
Given that, it's not surprising they'd be involved in funding and doing R&D on database systems.
PSA: Gaffer's tape makes the best ad-hoc mouse pad for surfaces that are not mousable... (at a trade show and your new shiny glass counters are acting weird with mice... make a small square with gaffers tape, which you already use to secure cords under carpet...
Exactly as @tomschlick says. Taking in lots of data, and analysing its internal relations (rather than analysing it based on imposed foreign key structure) is far easier in a graph DB and is better way of establishing links between data that would seem disparate in most other DB types.
It may used for targeting the drone assasination program, among other things. Given the who-talks-to-who metadata from a mass surveillance program, and a set of edge nodes manually identified as terrorists, graph analysis will tell you who's in the "centre" of that network of communications.
I've heard rumor that dead accounts like that can be taken over by someone else by emailing GitHub. A friend of mine did it, but I don't remember all the details of the situation
It took about two-hours from clicking the contact a human button to them releasing the name, they simply release it back and ask you to register quickly before someone else gets there first.
It is curious to me why GCHQ didn't just contact GitHub to acquire github.com/gchq but instead decided to go with the long and cumbersome github.com/GovernmentCommunicationsHeadquarters. Perhaps it is a British thing [1].
GitHub explicitly has a Name Squatting Policy [1] that states:
> Account names may not be inactively held for future use. GitHub account name squatting is prohibited. Inactive accounts may be renamed or removed by GitHub staff at their discretion.
I have been able to acquire a couple of GitHub names that have been inactive for several years by contacting GitHub support, and they usually reply within a day or so.
From [1]:
> Gaffer stores data in Accumulo, but inserting data and retrieving it again requires the user to have no knowledge of Accumulo. As Gaffer stores data in Accumulo, it is horizontally scalable so that very large data sets can be dealt with. It has an API that allows users to retrieve the data they care about, filtered according to their requirements and aggregated over the time window of interest. It supports bulk update and continuous update.
Seems like a very useful tool, especially if you already have accumulo infrastructure running. Docs need a bit more work I feel, but it's not terrible for a single page.
This is just a PR move to get the tech community to hate them less, even if only by a little bit. They want to muddy the waters and insert the idea into people's minds that "we're not all bad."
Not sure what you'd do with a tomahawk (besides hump it for fortitude).
Anyway...
In the US the policy is:
A United States government work is prepared by an officer or employee of the United States government as part of that person's official duties.
It is not subject to copyright in the United States and there are no copyright restrictions on reproduction, derivative works, distribution, performance, or display of the work. Anyone may, without restriction under U.S. copyright laws:
- reproduce the work in print or digital form
- create derivative works
- perform the work publicly
- display the work
- distribute copies or digitally transfer the work to the public by sale or other transfer of ownership, or by rental, lease, or lending.
Not that they're necessarily talking to each other about PR in the tech community, but looks like maybe the British Gov is trying to attract some talent and get devs engaged. Maybe they just want it improved for free :-p
I'd be interested in seeing the buy vs. build analysis for this project. Are there any pre-existing projects that have similar features? And assuming that this project is used to process classified data, what impact does this have on the selection process? e.g. is it possible to use closed-source solutions?
Just don't mention anything about a new JavaScript framework in any of your private communications, or they might spot it and make a new one of those too.
Get a [TinekerPop3](http://tinkerpop.incubator.apache.org) interface and I'll update my python libs to support it (one day...in the future...when I have time...and interest).
Do ends justify the means when it comes to knowledge being added to the world open-source repository of software? Should we, as a community, reject these people's hard work or just use it while also understanding that they're evil? I'm conflicted.
We can't just be consumers here, even though there is no money changing hands. This goes beyond the issue of trusting their code. We should send a clear message that 'dirty bits' are not welcome in community-built software. In effect, this is the only punishment you can dole out to an open source project--that is, choosing not to adopt it. Our 'ethics' as computer scientists are increasingly under fire and I think it's wise to know when to say 'no', especially when the hand that feeds is also the hand that beats you mercilessly.
It's simple, they are evil, so what they do has to be evil. They are evil because ... we are good ... and they obviously separate themselves by observing others, while hiding in secrecy. It's more complex than that, but here we go.
I feel as you do. As much as I am loathe to accept anything coming out of one of these organizations as an open-source project, I think it may actually do us more harm than good by attacking this project out of protest for the way they use their tools.
We can be as negative as we wish towards these agencies, but rejecting any and all attempts at communicating with the open source community is a strong way to reinforce their already insular culture. Embracing these projects in some way or another can possibly work as positive feedback toward greater organizational transparency, if not by the brass, than by the developers and engineers that work in these organizations.
"But software which OpenBSD uses and redistributes must be free
to all (be they people or companies), for any purpose they wish
to use it, including modification, use, peeing on, or even
integration into baby mulching machines or atomic bombs to be
dropped on Australia."
-- Theo de Raadt
Every large organisation has done some things that are morally wrong, so basing our reactions on the identity of the entire organisation is unworkable - too coarse-grained, too crude, to be anything but counterproductive. We need to base our reactions on actions and policies instead.
Thus, I applaud the helpful and constructive act of releasing this product as open source, and will certainly consider it if I ever need a graph database. This does not, of course, constitute approval of every GCHQ policy.
Emotionally, it feels to me a little like that one time a stalker bought me flowers and had them delivered to my (then-) home. I mean, yes, in general flowers are nice, but: fuck off! You can't buy my memories: a token of your affection won't make me forget what you did to me.
On reflection, the analogy bites a little closer than I might like to admit. They are stalkers, to each and every one of us. What they do is literally an attack on the entire internet (- IAB).
Please bear in mind that GCHQ are actually worse than the NSA in every way. They have essentially no "equities issue" to speak of; they operate both internationally and domestically; they have repeatedly ignored the law with essentially zero oversight, consequences or meaningful reproach; they have spied, and continue to spy, even on UK Government departments and MPs; and they are very probably about to get official powers to do mass hacking, which in typical form, they've already been doing for years anyway.
Bear in mind also that this is software that they use for analysis of data collected by spying on all of us; graph analysis software that is literally being used right now to select who to murder.
Forgive me if, therefore, I might hesitate to run any of the code of an organisation with a long history of deploying malware against innocent people.
Ethics, eh? So simple I don't know why people struggle with it.
Who are "these people"?
In what way are GCHQs coders and techies "evil"? Is it just because they have so widespread snooping powers? Is that still a problem if they have used those powers to prevent harm and injury from events that you won't have heard of? At what point does the latter outweigh the former?
I also suspect that if we follow the "GCHQ==Evil" logic, we would pretty quickly find that every coder working for a big enterprise is also "evil", and probably quite a few working for smaller ones too.
Given that it's OSS, it's not as if you're funding their vile actions by using it either.
The Snowden revelations showed unequivocally that the NSA surveillance did not help stop a SINGLE attack in the United States. All of the attacks were foiled due to regular people targeted intelligence. Forget about the ethics of surveilling millions of people indiscriminately, the sheer tax money wasted on this project alone is abhorrent.
And secondly, I'm sorry but i don't buy the slippery slope argument. As intelligent people we have clear boundaries about what is acceptable as the mandate of an organization and what isn't. If we took your approach to social issues we would never protest illegal wars because everyone else is involved in them or protest BP for polluting the Gulf because everyone drives cars. Its a ridiculous argument m
79 comments
[ 3.3 ms ] story [ 134 ms ] threadStill, tax payers money is used to write this stuff, good on them for open sourcing it.
The hypocrisy is staggering.
I just looked at the code: Java code that sits on top of the Hadoop file system. Supports date-binned data storage so it looks applicable to systems where you want to toss out old data occasionally.
oracle was founded to serve the CIA for fucks sake.
Given that, it's not surprising they'd be involved in funding and doing R&D on database systems.
https://en.wikipedia.org/wiki/Gaffer_%28filmmaking%29
Hence "Gaffer tape" which is black, rather than duct tape, which is the same construction but grey.
Its my favorite tape for general use.
I currently use this one on my glass desk: http://www.amazon.com/Logitech-Wireless-Anywhere-Mouse-Mac/d...
>Haha yeah I have at least ten of those... They do not work on certain glass surfaces, namely any shiny black granite desk.
That's specifically why I bought this one and it works flawlessly on every glass surface I've tried.
It's a bit harder to get this style, because when people say "gaffer tape" they normally mean "duck tape, or duct tape, or anything like that".
This guy is then killed and a press release put out about "Al-Quaeda #2 killed". http://www.longwarjournal.org/pakistan-strikes-hvts
Cybersquatters...
It took about two-hours from clicking the contact a human button to them releasing the name, they simply release it back and ask you to register quickly before someone else gets there first.
[1] https://www.addedbytes.com/blog/if-php-were-british/
> Account names may not be inactively held for future use. GitHub account name squatting is prohibited. Inactive accounts may be renamed or removed by GitHub staff at their discretion.
I have been able to acquire a couple of GitHub names that have been inactive for several years by contacting GitHub support, and they usually reply within a day or so.
[1] https://help.github.com/articles/name-squatting-policy/
I'd say they definitely have the skills to acquire it, if QUANTUM {INSERT|DNS}[1] are still operational. As you said, it's probably not worth it.
1. http://blog.fox-it.com/2015/04/20/deep-dive-into-quantum-ins...
Seems like a very useful tool, especially if you already have accumulo infrastructure running. Docs need a bit more work I feel, but it's not terrible for a single page.
[1]: https://github.com/GovernmentCommunicationsHeadquarters/Gaff...
I think it's more likely that it will help with recruitment, pull requests of high-quality might provide a potential interview or job offer.
No 'if', 'ands', or 'buts'. If it's paid for by the people it's the property of the people.
Anyway...
In the US the policy is:
A United States government work is prepared by an officer or employee of the United States government as part of that person's official duties.
It is not subject to copyright in the United States and there are no copyright restrictions on reproduction, derivative works, distribution, performance, or display of the work. Anyone may, without restriction under U.S. copyright laws:
- reproduce the work in print or digital form
- create derivative works
- perform the work publicly
- display the work
- distribute copies or digitally transfer the work to the public by sale or other transfer of ownership, or by rental, lease, or lending.
Source: https://www.usa.gov/government-works
US government based software developer agencies like 18f and USDS (United States Digital Service) license all of their code to the public domain.
The UK is slightly different: https://github.com/GovernmentCommunicationsHeadquarters/Gaff...
But there is the OGL (Open Government License): http://www.nationalarchives.gov.uk/information-management/re...
https://www.gchq-careers.co.uk/about-gchq.html
https://www.gchq-careers.co.uk/departments/technology-and-en...
The "applied research" is interesting: https://www.gchq-careers.co.uk/departments/applied-research....
I think the internet archive has some older pages with relevant information.
If this is GCHQ it's gently worrying - a github profile is pretty low on my list of what I want in the way of transparency and accountability.
As for my future software needs, I expect the code to be written at MI-6 and delivered just-in-time by James Bond.
We're being trolled.
Edit: Looking for forward to the release of RingTone, a system for processing very high volumes of call detail records in real time.
If you want a security expert's opinion, read Bruce Schneier's blog, and if you are inclined to learn more about the ethics, this page is great: http://cs.stanford.edu/people/eroberts/cs201/projects/ethics...
We can be as negative as we wish towards these agencies, but rejecting any and all attempts at communicating with the open source community is a strong way to reinforce their already insular culture. Embracing these projects in some way or another can possibly work as positive feedback toward greater organizational transparency, if not by the brass, than by the developers and engineers that work in these organizations.
Thus, I applaud the helpful and constructive act of releasing this product as open source, and will certainly consider it if I ever need a graph database. This does not, of course, constitute approval of every GCHQ policy.
On reflection, the analogy bites a little closer than I might like to admit. They are stalkers, to each and every one of us. What they do is literally an attack on the entire internet (- IAB).
Please bear in mind that GCHQ are actually worse than the NSA in every way. They have essentially no "equities issue" to speak of; they operate both internationally and domestically; they have repeatedly ignored the law with essentially zero oversight, consequences or meaningful reproach; they have spied, and continue to spy, even on UK Government departments and MPs; and they are very probably about to get official powers to do mass hacking, which in typical form, they've already been doing for years anyway.
Bear in mind also that this is software that they use for analysis of data collected by spying on all of us; graph analysis software that is literally being used right now to select who to murder.
Forgive me if, therefore, I might hesitate to run any of the code of an organisation with a long history of deploying malware against innocent people.
Who are "these people"?
In what way are GCHQs coders and techies "evil"? Is it just because they have so widespread snooping powers? Is that still a problem if they have used those powers to prevent harm and injury from events that you won't have heard of? At what point does the latter outweigh the former?
I also suspect that if we follow the "GCHQ==Evil" logic, we would pretty quickly find that every coder working for a big enterprise is also "evil", and probably quite a few working for smaller ones too.
Given that it's OSS, it's not as if you're funding their vile actions by using it either.
Oh well.
And secondly, I'm sorry but i don't buy the slippery slope argument. As intelligent people we have clear boundaries about what is acceptable as the mandate of an organization and what isn't. If we took your approach to social issues we would never protest illegal wars because everyone else is involved in them or protest BP for polluting the Gulf because everyone drives cars. Its a ridiculous argument m
Great to see spy agencies cooperating in open source :P