HN Explorer

Ask HN: As a Big Data engineer, how do I avoid helping governments spy on us?

20 points by pvnick ↗ HN
My career is based on data science, specifically building tools to gather, store, and make sense of large amounts of data. With the recent leaks regarding the NSA over the past couple days, I'm having a bit of an existential crisis so to speak. In fact I can actually envision how systems like PRISM could work and how to build them.

I love my work. It's fascinating and I believe that such powerful systems can do a lot of good. I'm specifically trying to get into bio- and cheminformatics to try and develop technologies that save lives and improve quality of life, but there is a lot of overlap between the technology I develop for that purpose and what Silicon Valley uses to collect innocent citizen data and pass it off to the NSA.

My question is, how do I go about ensuring that my work is only used for the benefit of mankind? Is that possible within my career? Dystopian literature has always fascinated me in a sort of macabre sense, but I absolutely loath the idea of contributing to the technology that oppressive governments would use to spy on their citizens.

Can HN give me some advice?

4 comments

[ 285 ms ] story [ 341 ms ] thread
[dead] fuuuuuuuuuuu
There are technical papers on how to have anonymity preserved against questions to large data sets - read them.

The gist is that questions should only be possible about aggregate facts - virtually any question which can give fine grained information about the existence or facts of a single user with a high degree of accuracy can be used to strip away privacy.

Further, there's only so much privacy a dataset has. This means that there's a limited number of questions you can ask it and still expect people to maintain reasonable privacy. (Has a technical meaning in papers.)

So my advice on what to do now is simple: design your database such that it scrambles user data when it's entered in a way that horribly mangles questions about individuals, but preserves the distribution, hence answers about aggregate facts. (If you're looking at disease rates or similar, this should be sufficient.)

If this isn't possible, why not just ask people for private information? People are often willing to share, at least for a good cause.

The key point of all of this is that privacy in a data set must be regarded as a commodity: once it's used up, you have to move on to a new data set or you've given up all the privacy of your users. (Once the threshold is crossed, privacy is rapidly lost against malicious questions.)

Looking towards the future, some kind of holomorphic encryption scheme on the data set, effectively only allowing you to ask certain questions without destroying the data, and enforcing such policy at a technical level (unless you can manage to decrypt the data/sign your own programs/etc).

basdevries
I don't think you can do more than just doing what you do. You can aware customers on the front-end that they arn't supposed to save any data on your system they are afraid of sharing. I can understand this is difficult from an ethical perspective but I think that as long as you arn't actually stealing data or participating in systems like PRISM you should just be fine.
gesman
If you'll shift your attention to being opinionated about politics and taking sides "for" and "against" - your valuable skills will stop growing.

Just keep doing what you love and things will resolve. World is not broken.

[deleted]
(comment deleted)